API to obtain metadata of all ITU recommendations?

[strogonoff]

This may be helpful since we’d be able to obtain recommendation details (link, publication data, status, etc.) automatically, letting authors only specify the code.

For example, author writes E.164 and we will be able to fill in the date or other details as requested.

[strogonoff]

So far the only way I can see is scraping URLs of the form https://www.itu.int/rec/T-REC-{{ recommendation code }}/en, but I don’t know if this is viable for all recommendations and whether this webpage is deprecated (I didn’t navigate there using ITU site, only discovered from Google).

[ronaldtse]

@strogonoff the mechanism already exists, in https://github.com/metanorma/relaton-itu 😉

The relaton gem is able to provide YAML output for every ITU Recommendation.

[strogonoff]

@ronaldtse That’s glorious!

[strogonoff]

@ronaldtse It looks like relaton-itu can look up individual recommendations, but from what I could see it’s not capable of generating a list of all ITU recommendations (or, say, all ITU recommendations approved within given timeframe). Which would natural, since it’s a bibliography library that parses specific references.

In perfect world we would have an ITU recommendation API—which we could actually build ourselves, starting with scraper logic in relaton-itu. Then relaton-itu could even use this API and simplify its logic, especially if we can get ITU to officially support/recognize the API.

[ronaldtse]

it’s not capable of generating a list of all ITU recommendations (or, say, all ITU recommendations approved within given timeframe). Which would natural, since it’s a bibliography library that parses specific references.

You're asking for a lot 😉 Hmm... But the ITU publications interface doesn't allow us to find by dates.

In perfect world we would have an ITU recommendation API—which we could actually build ourselves, starting with scraper logic in relaton-itu. Then relaton-itu could even use this API and simplify its logic, especially if we can get ITU to officially support/recognize the API.

Yes we could do it, it would actually be a great case for Relaton.

But how do we get the data...? It's going to be a nightmare to integrate with their existing document system...

[strogonoff]

@ronaldtse

But how do we get the data...? It's going to be a nightmare to integrate with their existing document system...

What I thought:

• we could borrow the individual recommendation processing logic from Relaton, which is done by scraping,
• extend it to parsing recommendation lists, also by scraping,
• having obtained a dataset of recommendations, wrap this into a neat API and offer querying possibilities,
• get ITU to recognize and support the API[0]
• use the resources for API support to evolve beyond scraping and eventually normalize their document system a bit

[0] this is the big one. Obviously having an API enables various unexpected uses and opens possibilities, but that’s a difficult point to get across. Beyond that abstract idea, we don’t have many concrete use cases to show. If we want bibliography, they’d say we already have it through Relaton’s scraping; and beyond that we only want it for OB generation