Architecture of Issue Importer 2.0

[karlcow]

@miketaylr So today I was working on the code and there were two many issues which were popping up with our discussions of the last few days. So before refactoring wildly and destroying everything that mike did. I want to know if it makes sense. ;)

Given a list of files with a JSON format containing each of them an issue,
A user may pipe the list of files to the issue importer to post them on Github.
(When I say webcompat below I may meant github in the backend)

1. Grab the list of labels from webcompat, save them locally in a cache.
2. For each file in the filelist:
   1. read the file content as JSON
   2. transform the JSON content to webcompat JSON with the relevant adapter [origin]
      1. Select the appropriate data
      2. Add the appropriate labels
      3. Generate the JSON
   3. check the validity
      1. If invalid: log the issue numbers in logfile and discard the file
      2. else: add the file to a batch folder.
3. If github true:
   1. Ask confirmation that you are about to upload 1 million issues.
   2. For each of the file in the batch folder
      1. Upload the file to github
      2. Delete from the batch. (If there is an issue in the middle, you do not have to redo everything)
   3. Make a summary of what happened
4. else: (dry-run):
   1. Give a summary of what it would have done without doing it.

[karlcow]

To reduce the complexity, we could impose an input format. In this case, no conversion at all, just checking the validity. And it also means that we do not need adapters at all.

[hallvors]

Looks good to me - although one light "adapter" script and one light "importer" script would also be usable? So one script which reads Bugzilla JSON data and outputs JSON data for webcompat, and one that just reads a JSON file and pushes data into the webcompat DB? The main question is where to add label logic - some of the labels depend on source data and maybe some can be deduced with some heuristics in the importer?

[miketaylr]

To reduce the complexity, we could impose an input format. In this case, no conversion at all, just checking the validity. And it also means that we do not need adapters at all.

That's what we have right now, no? So we're done.

although one light "adapter" script and one light "importer" script would also be usable? So one script which reads Bugzilla JSON data and outputs JSON data for webcompat, and one that just reads a JSON file and pushes data into the webcompat DB?

OK, now we're half done. :p

I think what you have in the first comment looks good to me if you want to take that approach. And if you think it's too complex we make a (temporary?) judgement call that you can only import a single issue at a time. Which is probably safer in terms of accidentally importing a bunch of unwanted things.

Whichever approach you decide in the end, let's develop it in a branch so we can change our minds in the future. :)

[karlcow]

That's what we have right now, no? So we're done.

hehe. Nice try. We only handle one issue and not multiple issues. ^_~ But yes much simpler.

So one script which reads Bugzilla JSON data

This is already the case. In some fashion, Give a list of bug numbers or generate manually one bug and then save it as JSON.

let's develop it in a branch so we can change our minds in the future. :)

This is the case already for the bugzilla adapter branch.
https://github.com/karlcow/issue-importer/commits/moz-adapter

So yes a new branch would be necessary.