Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use needles from correct ref of NEEDLES_DIR #5175

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions etc/openqa/openqa.ini
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,10 @@
#git_auto_clone = yes
## Experimental - Ensure a git update of all test code and needles
#git_auto_update = no
# whether openQA should attempt to display needles of the correct version in the web UI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a feeling that the comment is a bit cryptic from users' side. What does it mean attempt? Also correct version I think is not very explanatory. I am talking from the POV of someone who do not know much about the openQA or a new user.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we'll be able to explain this feature within the config file. I think this comment is good enough for people trying to enable this feature after reading documentation about it elsewhere. So we could document this feature as part of the normal openQA documentation. However, I'd be fine keeping this as an almost undocumented "expert feature" for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, maybe we can still change the word "correct". Who wouldn't want "correct versions"? :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about

Suggested change
# whether openQA should attempt to display needles of the correct version in the web UI
# whether openQA should display needles from different branches in the web UI

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not about the branch that was used (branches are moving targets) but really about the exact version that was used (not a moving target). This lookup will probably also be best effort. So I find the version of this comment as it is right now much better than the suggested change.

Who wouldn't want "correct versions"? :)

Someone who doesn't want to pay the price for it. We could state the disadvantages here, especially that the feature is still experimental.

#checkout_needles_sha = no
# retention for storing temporary needle refs in minutes
#temp_needle_refs_retention = 120

## Authentication method to use for user management
[auth]
Expand Down
13 changes: 13 additions & 0 deletions lib/OpenQA/Git.pm
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
use Mojo::Base -base, -signatures;
use Mojo::Util 'trim';
use Cwd 'abs_path';
use Mojo::File 'path';
use OpenQA::Utils qw(run_cmd_with_log_return_error);

has 'app';
Expand Down Expand Up @@ -145,4 +146,16 @@
return $r->{status};
}

sub cache_ref ($self, $ref, $relative_path, $output_file) {
if (-f $output_file) {
eval { path($output_file)->touch };
return $@ ? $@ : undef;

Check warning on line 152 in lib/OpenQA/Git.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Git.pm#L149-L152

Added lines #L149 - L152 were not covered by tests
}
my @git = $self->_prepare_git_command;
my $res = run_cmd_with_log_return_error [@git, 'show', "$ref:./$relative_path"], output_file => $output_file;
return undef if $res->{status};
unlink $output_file;
return _format_git_error($res, 'Unable to cache Git ref');

Check warning on line 158 in lib/OpenQA/Git.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Git.pm#L154-L158

Added lines #L154 - L158 were not covered by tests
}

1;
57 changes: 57 additions & 0 deletions lib/OpenQA/Needles.pm
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Copyright SUSE LLC
# SPDX-License-Identifier: GPL-2.0-or-later

package OpenQA::Needles;
use Mojo::Base -strict, -signatures;

use Exporter qw(import);
use File::Basename;
use File::Spec;
use File::Spec::Functions qw(catdir);
use OpenQA::Git;
use OpenQA::Log qw(log_error);
use OpenQA::Utils qw(prjdir sharedir);
use Mojo::File qw(path);

our @EXPORT = qw(temp_dir is_in_temp_dir needle_temp_dir locate_needle);

my $tmp_dir = prjdir() . '/webui/cache/needle-refs';

sub temp_dir () { $tmp_dir }

sub is_in_temp_dir ($file_path) { index($file_path, $tmp_dir) == 0 }

sub needle_temp_dir ($dir, $ref) { path($tmp_dir, basename(dirname($dir)), $ref, 'needles') }

Check warning on line 24 in lib/OpenQA/Needles.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Needles.pm#L24

Added line #L24 was not covered by tests

sub _locate_needle_for_ref ($relative_needle_path, $needles_dir, $needles_ref) {
return undef unless defined $needles_ref;

my $temp_needles_dir = needle_temp_dir($needles_dir, $needles_ref);
my $subdir = dirname($relative_needle_path);
path($temp_needles_dir, $subdir)->make_path if File::Spec->splitdir($relative_needle_path) > 1;

Check warning on line 31 in lib/OpenQA/Needles.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Needles.pm#L29-L31

Added lines #L29 - L31 were not covered by tests

my $git = OpenQA::Git->new(dir => $needles_dir);
my $temp_json_path = "$temp_needles_dir/$relative_needle_path";
my $basename = basename($relative_needle_path, '.json');
my $relative_png_path = "$subdir/$basename.png";
my $temp_png_path = "$temp_needles_dir/$relative_png_path";
my $error = $git->cache_ref($needles_ref, $relative_needle_path, $temp_json_path)

Check warning on line 38 in lib/OpenQA/Needles.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Needles.pm#L33-L38

Added lines #L33 - L38 were not covered by tests
// $git->cache_ref($needles_ref, $relative_png_path, $temp_png_path);
return $temp_json_path unless defined $error;
log_error "An error occurred when looking for ref '$needles_ref' of '$relative_needle_path': $error";
return undef;

Check warning on line 42 in lib/OpenQA/Needles.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/Needles.pm#L40-L42

Added lines #L40 - L42 were not covered by tests
}

sub locate_needle ($relative_needle_path, $needles_dir, $needles_ref = undef) {
my $location_for_ref = _locate_needle_for_ref($relative_needle_path, $needles_dir, $needles_ref);
return $location_for_ref if defined $location_for_ref;
my $absolute_filename = catdir($needles_dir, $relative_needle_path);
my $needle_exists = -f $absolute_filename;
if (!$needle_exists) {
$absolute_filename = catdir(sharedir(), $relative_needle_path);
$needle_exists = -f $absolute_filename;
}
return $absolute_filename if $needle_exists;
log_error "Needle file $relative_needle_path not found within $needles_dir.";
return undef;
}
2 changes: 1 addition & 1 deletion lib/OpenQA/Schema/Result/Needles.pm
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ use OpenQA::App;
use OpenQA::Git;
use OpenQA::Jobs::Constants;
use OpenQA::Schema::Result::Jobs;
use OpenQA::Utils qw(locate_needle);
use OpenQA::Needles qw(locate_needle);

__PACKAGE__->table('needles');
__PACKAGE__->load_components(qw(InflateColumn::DateTime Timestamps));
Expand Down
2 changes: 2 additions & 0 deletions lib/OpenQA/Setup.pm
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ sub read_config ($app) {
do_cleanup => 'no',
git_auto_clone => 'yes',
git_auto_update => 'no',
checkout_needles_sha => 'no',
temp_needle_refs_retention => 120,
},
scheduler => {
max_job_scheduled_time => 7,
Expand Down
5 changes: 4 additions & 1 deletion lib/OpenQA/Shared/Plugin/Gru.pm
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ sub register_tasks ($self) {
OpenQA::Task::Asset::Download
OpenQA::Task::Asset::Limit
OpenQA::Task::Git::Clone
OpenQA::Task::Needle::Scan OpenQA::Task::Needle::Save OpenQA::Task::Needle::Delete
OpenQA::Task::Needle::Scan
OpenQA::Task::Needle::Save
OpenQA::Task::Needle::Delete
OpenQA::Task::Needle::LimitTempRefs
OpenQA::Task::Job::Limit
OpenQA::Task::Job::ArchiveResults
OpenQA::Task::Job::FinalizeResults
Expand Down
39 changes: 39 additions & 0 deletions lib/OpenQA/Task/Needle/LimitTempRefs.pm
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Copyright SUSE LLC
# SPDX-License-Identifier: GPL-2.0-or-later

package OpenQA::Task::Needle::LimitTempRefs;
use Mojo::Base 'Mojolicious::Plugin', -signatures;

use File::Find;
use File::stat;
use Fcntl qw(S_ISDIR);
use OpenQA::Needles;
use OpenQA::Task::SignalGuard;
use Time::Seconds;

my $retention;

sub register ($self, $app, $job) {
$retention = $app->config->{'scm git'}->{temp_needle_refs_retention} * ONE_MINUTE;
$app->minion->add_task(limit_temp_needle_refs => sub ($job) { _limit($app, $job) });
}

sub _limit ($app, $job) {
my $ensure_task_retry_on_termination_signal_guard = OpenQA::Task::SignalGuard->new($job);

return $job->finish({error => 'Another job to remove needle versions is running. Try again later.'})
unless my $guard = $app->minion->guard('limit_needle_versions_task', 7200);

# remove all temporary needles which haven't been accessed in time period specified in config
my $temp_dir = OpenQA::Needles::temp_dir;
return undef unless -d $temp_dir;
my $now = time;
my $wanted = sub {
return undef unless my $lstat = lstat $File::Find::name;
return rmdir $File::Find::name if S_ISDIR($lstat->mode); # remove all empty dirs
return unlink $File::Find::name if ($now - $lstat->mtime) > $retention;
};
find({no_chdir => 1, bydepth => 1, wanted => $wanted}, $temp_dir);
}

1;
22 changes: 4 additions & 18 deletions lib/OpenQA/Utils.pm
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ use OpenQA::Log qw(log_info log_debug log_warning log_error);
use Config::Tiny;
use Time::HiRes qw(tv_interval);
use File::Basename;
use File::Path qw(make_path);
use File::Spec;
use File::Spec::Functions qw(catfile catdir);
use Fcntl;
Expand Down Expand Up @@ -92,7 +93,6 @@ our @EXPORT = qw(
BUGREF_REGEX
LABEL_REGEX
FLAG_REGEX
locate_needle
needledir
productdir
testcasedir
Expand Down Expand Up @@ -254,22 +254,6 @@ sub is_in_tests {

sub needledir { productdir(@_) . '/needles' }

sub locate_needle {
my ($relative_needle_path, $needles_dir) = @_;

my $absolute_filename = catdir($needles_dir, $relative_needle_path);
my $needle_exists = -f $absolute_filename;

if (!$needle_exists) {
$absolute_filename = catdir(sharedir(), $relative_needle_path);
$needle_exists = -f $absolute_filename;
}
return $absolute_filename if $needle_exists;

log_error("Needle file $relative_needle_path not found within $needles_dir.");
return undef;
}

# Adds a timestamp to a string (eg. needle name) or replace the already present timestamp
sub ensure_timestamp_appended {
my ($str) = @_;
Expand Down Expand Up @@ -333,10 +317,12 @@ sub run_cmd_with_log {
sub run_cmd_with_log_return_error ($cmd, %args) {
my $stdout_level = $args{stdout} // 'debug';
my $stderr_level = $args{stderr} // 'debug';
my $output_file = $args{output_file};
log_info('Running cmd: ' . join(' ', @$cmd));
try {
my ($stdin, $stdout_err, $stdout, $stderr) = ('') x 4;
my $ipc_run_succeeded = IPC::Run::run($cmd, \$stdin, \$stdout, \$stderr);
my @out_args = defined $output_file ? ('>', $output_file, '2>', \$stderr) : (\$stdout, \$stderr);
my $ipc_run_succeeded = IPC::Run::run($cmd, \$stdin, @out_args);
my $return_code = $?;
chomp $stderr;
if ($ipc_run_succeeded) {
Expand Down
14 changes: 13 additions & 1 deletion lib/OpenQA/WebAPI/Controller/File.pm
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

BEGIN { $ENV{MAGICK_THREAD_LIMIT} = 1; }

use OpenQA::Needles;
use OpenQA::Utils qw(:DEFAULT prjdir assetdir imagesdir);
use File::Basename;
use File::Spec;
Expand All @@ -29,11 +30,22 @@

# make sure the directory of the file parameter is a real subdir of testcasedir before
# using it to find needle subdirectory, to prevent access outside of the zoo
if ($jsonfile && !is_in_tests($jsonfile)) {
if ($jsonfile && !is_in_tests($jsonfile) && !OpenQA::Needles::is_in_temp_dir($jsonfile)) {
my $prjdir = prjdir();
warn "$jsonfile is not in a subdir of $prjdir/share/tests or $prjdir/tests";
return $self->render(text => 'Forbidden', status => 403);
}
# If the json file in not in the tests we may be using a temporary
# directory for needles from a different git SHA
my $jsonfile_in_temp_dir = $jsonfile && OpenQA::Needles::is_in_temp_dir($jsonfile);
if ($jsonfile_in_temp_dir) {
$needledir = dirname($jsonfile);

Check warning on line 42 in lib/OpenQA/WebAPI/Controller/File.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/WebAPI/Controller/File.pm#L42

Added line #L42 was not covered by tests
# In case we're in a subdirectory, keep taking the dirname until we
# have the path of the `needles` directory
while (basename($needledir) ne 'needles') {
$needledir = dirname($needledir);

Check warning on line 46 in lib/OpenQA/WebAPI/Controller/File.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/WebAPI/Controller/File.pm#L45-L46

Added lines #L45 - L46 were not covered by tests
}
}
# Reject directory traversal breakouts here...
if (index($jsonfile, '..') != -1) {
warn "jsonfile value $jsonfile is invalid, cannot contain ..";
Expand Down
64 changes: 51 additions & 13 deletions lib/OpenQA/WebAPI/Controller/Step.pm
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@
use Mojo::File 'path';
use Mojo::URL;
use Mojo::Util 'decode';
use OpenQA::Utils qw(ensure_timestamp_appended find_bug_number locate_needle needledir testcasedir);
use OpenQA::Needles qw(needle_temp_dir locate_needle);
use OpenQA::Utils qw(ensure_timestamp_appended find_bug_number needledir testcasedir
run_cmd_with_log run_cmd_with_log_return_error);
use OpenQA::Jobs::Constants;
use File::Basename;
use File::Path 'make_path';
use File::Which 'which';
use POSIX 'strftime';
use Mojo::JSON 'decode_json';
Expand Down Expand Up @@ -91,6 +94,34 @@
$self->viewimg;
}

sub _determine_needles_dir_for_job ($self, $job) {
return undef unless $self->app->config->{'scm git'}->{checkout_needles_sha} eq 'yes';
return undef unless my $needle_dirs = realpath($job->needle_dir);
my $settings = $job->settings;
return ($needle_dirs, $settings->single({key => 'NEEDLES_DIR'}) // $settings->single({key => 'CASEDIR'}));

Check warning on line 101 in lib/OpenQA/WebAPI/Controller/Step.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/WebAPI/Controller/Step.pm#L99-L101

Added lines #L99 - L101 were not covered by tests
}

sub _create_tmpdir_for_needles_refspec ($self, $job) {
my ($needle_dirs, $needles_dir_var) = $self->_determine_needles_dir_for_job($job);
return undef unless $needles_dir_var;
my $needles_url = Mojo::URL->new($needles_dir_var->value);
return undef unless $needles_url->scheme;
my $needles_ref = $needles_url->fragment;
eval {
my $vars = decode_json(path($job->result_dir, 'vars.json')->slurp);
$needles_ref = $vars->{NEEDLES_GIT_HASH} if ref $vars eq 'HASH';

Check warning on line 112 in lib/OpenQA/WebAPI/Controller/Step.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/WebAPI/Controller/Step.pm#L107-L112

Added lines #L107 - L112 were not covered by tests
};
chomp $needles_ref;
return undef unless $needles_ref;
return undef unless run_cmd_with_log ['git', '-C', $needle_dirs, 'fetch', '--depth', 1, 'origin', $needles_ref];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sdclarke We have not forgotten about this.

We are currently working on some git features and fixes, and I just wanted to note that we have a git_auto_clone feature now that will fetch any requested ref on the webui (branch or sha) in a minion job before the job starts running. (Note that it only works if it is the same origin and not some fork.)

That would make this fetch here not necessary anymore and also avoid the need for a lockfile.
The sha should always be in NEEDLES_GIT_HASH so the rev-parse also seems unnecessary.

Would you consider to use the git_auto_clone feature in order to not have to implement that part here?

I'm not sure if there are any cases which git_auto_clone wouldn't cover. @Martchus said there might be.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that git_auto_clone only helps if a Git-URL is specified as NEEDLES_DIR (or CASEDIR which is relevant if needles are part of the tests).

Even then it is not guaranteed to fetch the refs the test actually runs on. The test might specify a branch name (or no branch which means "default branch") and when the test is finally executed on the worker host the ref that branch points to might differ from the ref checked out by the web UI at the time the test was scheduled.

Additionally, we probably want to have some cleanup for the fetching done by the git_auto_clone feature. So the ref might have also already been cleaned up.

So I think we need that fetch in any case.


That would make this fetch here not necessary anymore and also avoid the need for a lockfile.

I don't think we need a lockfile here. We can probably arrange it so that it would not matter if any other commands are executed in the middle. (The use of FETCH_HEAD definitely needs to go way but it can probably be replaced with whatever we specify on the fetch command.)

The sha should always be in NEEDLES_GIT_HASH so the rev-parse also seems unnecessary.

That's true. So we could (and probably should) get rid of it. If one specifies a branch name via NEEDLES_DIR the NEEDLES_GIT_HASH still has the concrete sha.

Copy link
Contributor

@perlpunk perlpunk Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that git_auto_clone only helps if a Git-URL is specified as NEEDLES_DIR (or CASEDIR which is relevant if needles are part of the tests).

  • The PR description says that it is about CASEDIR/NEEDLES_DIR being set.
  • And if it is not set, there is no certain ref to fetch, just the default remote branch
  • In the near future, git_auto_clone will also automatically update clones if CASEDIR/NEEDLES_DIR is empty. It will just update the checkout branch in the checked out directory.

And this is all we need, no?
What else needs to be fetched that git_auto_clone does not fetch?

The only thing could be forks, but as far as I can see this PR also doesn't handle forks.

Even then it is not guaranteed to fetch the refs the test actually runs on. The test might specify a branch name (or no branch which means "default branch") and when the test is finally executed on the worker host the ref that branch points to might differ from the ref checked out by the web UI at the time the test was scheduled.

What's checked out on the webui does not matter. The important thing is that it has been fetched at some point, so we can do a git show on it, right?

Additionally, we probably want to have some cleanup for the fetching done by the git_auto_clone feature. So the ref might have also already been cleaned up.

Yes. I'm not sure what git already offers there and how to know what we can cleanup.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's checked out on the webui does not matter. The important thing is that it has been fetched at some point, so we can do a git show on it, right?

I meant "fetched by the web UI". And I meant the following sequence of events:

  1. A test is scheduled pointing to a branch, e.g. just the default branch.
  2. The web UI fetches the latest ref of that branch because git_auto_clone = 1 is configured.
  3. Someone pushes additional commits to that branch or even force-pushes there.
  4. The test is finally assigned to a worker and that worker now fetches a different ref than the web UI in step 2 so NEEDLES_GIT_HASH points to a ref that is not yet checked out (despite git_auto_clone = 1).
  5. Someone wants to display the candidate needles for that job. We need to fetch the missing ref.

And this is not very contrived. That someone specifies a branch where many commits happen (e.g. the default branch of some repo with many contributors) is probably a common use case. That it can take between the scheduling of a job and the time it actually gets executed is also something we need to expect.

my $rev_parse_res = run_cmd_with_log_return_error ['git', '-C', $needle_dirs, 'rev-parse', 'FETCH_HEAD'];
return undef unless $rev_parse_res->{status};
$needles_ref = $rev_parse_res->{stdout};
chomp $needles_ref;
needle_temp_dir($needle_dirs, $needles_ref)->make_path;
return $needles_ref;

Check warning on line 122 in lib/OpenQA/WebAPI/Controller/Step.pm

View check run for this annotation

Codecov / codecov/patch

lib/OpenQA/WebAPI/Controller/Step.pm#L114-L122

Added lines #L114 - L122 were not covered by tests
}

# Needle editor
sub edit ($self) {
return $self->reply->not_found unless $self->_init && $self->check_tabmode();
Expand All @@ -101,6 +132,7 @@
my $distri = $job->DISTRI;
my $dversion = $job->VERSION || '';
my $needle_dir = $job->needle_dir;
my $needle_ref = $self->_create_tmpdir_for_needles_refspec($job);
my $app = $self->app;
my $needles_rs = $app->schema->resultset('Needles');

Expand Down Expand Up @@ -130,7 +162,7 @@
# Second position: the only needle (with the same matches)
my $needle_info
= $self->_extended_needle_info($needle_dir, $needle_name, \%basic_needle_data, $module_detail->{json},
0, \@error_messages);
0, \@error_messages, $needle_ref);
if ($needle_info) {
$needle_info->{matches} = $screenshot->{matches};
push(@needles, $needle_info);
Expand All @@ -144,10 +176,10 @@
# $needle contains information from result, in which 'areas' refers to the best matches.
# We also use $area for transforming the match information into a real area
for my $needle (@$module_detail_needles) {
my $needle_info = $self->_extended_needle_info(
$needle_dir, $needle->{name}, \%basic_needle_data,
$needle->{json}, $needle->{error}, \@error_messages
) || next;
my $needle_info
= $self->_extended_needle_info($needle_dir, $needle->{name}, \%basic_needle_data,
$needle->{json}, $needle->{error}, \@error_messages, $needle_ref)
|| next;
my $matches = $needle_info->{matches};
for my $match (@{$needle->{area}}) {
my %area = (
Expand Down Expand Up @@ -188,7 +220,7 @@
# get needle info to show the needle also in selection
my $needle_info
= $self->_extended_needle_info($needle_dir, $new_needle->name, \%basic_needle_data, $new_needle->path,
undef, \@error_messages)
undef, \@error_messages, $needle_ref)
|| next;
$needle_info->{title} = 'new: ' . $needle_info->{title};
push(@needles, $needle_info);
Expand Down Expand Up @@ -274,9 +306,9 @@
return \%screenshot;
}

sub _basic_needle_info ($self, $name, $distri, $version, $file_name, $needles_dir) {
sub _basic_needle_info ($self, $name, $distri, $version, $file_name, $needles_dir, $needle_ref) {
$file_name //= "$name.json";
$file_name = locate_needle($file_name, $needles_dir) if !-f $file_name;
$file_name = locate_needle($file_name, $needles_dir, $needle_ref) if !-f $file_name;
return (undef, 'File not found') unless defined $file_name;

my $needle;
Expand All @@ -303,11 +335,14 @@
return ($needle, undef);
}

sub _extended_needle_info ($self, $needle_dir, $needle_name, $basic_needle_data, $file_name, $error, $error_messages) {
sub _extended_needle_info ($self, $needle_dir, $needle_name, $basic_needle_data, $file_name, $error, $error_messages,
$needle_ref)
{
my $overall_list_of_tags = $basic_needle_data->{tags};
my $distri = $basic_needle_data->{distri};
my $version = $basic_needle_data->{version};
my ($needle_info, $err) = $self->_basic_needle_info($needle_name, $distri, $version, $file_name, $needle_dir);
my ($needle_info, $err)
= $self->_basic_needle_info($needle_name, $distri, $version, $file_name, $needle_dir, $needle_ref);
unless (defined $needle_info) {
push(@$error_messages, "Could not parse needle $needle_name for $distri $version: $err");
return undef;
Expand Down Expand Up @@ -467,6 +502,7 @@
my $distri = $job->DISTRI;
my $dversion = $job->VERSION || '';
my $needle_dir = $job->needle_dir;
my $needle_ref = $self->_create_tmpdir_for_needles_refspec($job);
my $real_needle_dir = realpath($needle_dir) // $needle_dir;
my $needles_rs = $self->app->schema->resultset('Needles');

Expand Down Expand Up @@ -502,7 +538,8 @@
# load primary needle match
my $primary_match;
if (my $needle = $module_detail->{needle}) {
my ($needleinfo) = $self->_basic_needle_info($needle, $distri, $dversion, $module_detail->{json}, $needle_dir);
my ($needleinfo)
= $self->_basic_needle_info($needle, $distri, $dversion, $module_detail->{json}, $needle_dir, $needle_ref);
if ($needleinfo) {
my $info = {
name => $needle,
Expand All @@ -524,7 +561,8 @@
if ($module_detail->{needles}) {
for my $needle (@{$module_detail->{needles}}) {
my $needlename = $needle->{name};
my ($needleinfo) = $self->_basic_needle_info($needlename, $distri, $dversion, $needle->{json}, $needle_dir);
my ($needleinfo)
= $self->_basic_needle_info($needlename, $distri, $dversion, $needle->{json}, $needle_dir, $needle_ref);
next unless $needleinfo;
my $info = {
name => $needlename,
Expand Down
2 changes: 2 additions & 0 deletions script/openqa-enqueue-needle-ref-cleanup
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/sh -e
exec "$(dirname "$0")"/openqa eval -m production -V 'app->gru->enqueue(limit_temp_needle_refs => [], {priority => 5, ttl => 172800, limit => 1})' "$@"
Loading
Loading