+
+
[](https://cran.r-project.org/package=eurlex)
[](https://cran.r-project.org/package=eurlex)
[](https://github.com/michalovadek/eurlex/actions/workflows/check-standard.yaml)
-# eurlex
+
-The `eurlex` R package attempts to significantly reduce the overhead associated with using SPARQL and REST APIs made available by the EU Publication Office and other EU institutions. Compared to pure web-scraping, the package provides more efficient and transparent access to data on European Union laws and policies.
+The `eurlex` R [package](https://michalovadek.github.io/eurlex/) reduces the overhead associated with using SPARQL and REST APIs made available by the EU Publication Office and other EU institutions. Compared to pure web-scraping, the package provides more efficient and transparent access to data on European Union laws and policies.
-See the [vignette](https://michalovadek.github.io/eurlex/articles/eurlexpkg.html) for a walkthrough on how to use the package. Check function documentation for most up-to-date overview of features. Example use cases are shown in this [paper](https://www.tandfonline.com/doi/full/10.1080/2474736X.2020.1870150).
+See the [vignette](https://michalovadek.github.io/eurlex/articles/eurlexpkg.html) for a basic walkthrough on how to use the package. Check function documentation for most up-to-date overview of features. Example use cases are shown in this [paper](https://www.tandfonline.com/doi/full/10.1080/2474736X.2020.1870150).
You can use `eurlex` to create automatically updated overviews of EU decision-making activity, as shown [here](https://michalovadek.github.io/eulaw/).
@@ -18,31 +21,39 @@ The development version is available via `remotes::install_github("michalovadek/
Michal Ovádek (2021) **Facilitating access to data on European Union laws**, *Political Research Exchange*, 3:1, DOI: [10.1080/2474736X.2020.1870150](https://www.tandfonline.com/doi/full/10.1080/2474736X.2020.1870150)
## Basic usage
-
The `eurlex` package currently envisions the typical use-case to consist of getting bulk information about EU legislation into R as fast as possible. The package contains three core functions to achieve that objective: `elx_make_query()` to create pre-defined or customized SPARQL queries; `elx_run_query()` to execute the pre-made or any other manually input query; and `elx_fetch_data()` to fire GET requests for certain metadata to the REST API.
The function `elx_make_query` takes as its first argument the type of resource to be retrieved (such as "directive" or "any") from the semantic database that powers Eur-Lex (and other publications) called Cellar. If you are familiar with SPARQL, you can always specify your own queries and execute them with `elx_run_query()`.
-`elx_run_query()` executes SPARQL queries on a pre-specified endpoint of the EU Publication Office. It outputs a `data.frame` where each column corresponds to one of the requested variables, while the rows accumulate observations of the resource type satisfying the query criteria. Obviously, the more data is to be returned, the longer the execution time, varying from a few seconds to several hours, depending also on your connection. The first column always contains the unique URI of a "work" (legislative act or court judgment) which identifies each resource in Cellar. Several human-readable identifiers are normally associated with each "work" but the most useful one is [CELEX](https://eur-lex.europa.eu/content/tools/TableOfSectors/types_of_documents_in_eurlex.html), retrieved by default.
+`elx_run_query()` executes SPARQL queries on a pre-specified endpoint of the EU Publication Office. It outputs a `data.frame` where each column corresponds to one of the requested variables, while the rows accumulate observations of the resource type satisfying the query criteria. Obviously, the more data is to be returned, the longer the execution time, varying from a few seconds to several hours, depending also on your connection. The first column always contains the unique URI of a "work" (usually legislative act or court judgment) which identifies each resource in Cellar. Several human-readable identifiers are normally associated with each "work" but the most useful one tends to be [CELEX](https://eur-lex.europa.eu/content/tools/TableOfSectors/types_of_documents_in_eurlex.html), retrieved by default.
-For the moment, it is recommended to retrieve metadata one variable at a time. For example, if you wish to obtain the legal bases of directives and the date of transposition, you should run separate calls:
+``` r
+# load library
+library(eurlex)
-0. `ids <- elx_make_query("directive") |> elx_run_query()`
-1. `lbs <- elx_make_query("directive", include_lbs = TRUE) |> elx_run_query()`
-2. `dates <- elx_make_query("directive", include_date_transpos = TRUE) |> elx_run_query()`
-3. `ids |> dplyr::left_join(lbs) |> dplyr::left_join(dates)`
+# create query
+query <- elx_make_query("directive", include_date_transpos = TRUE)
-rather than `elx_make_query("directive", include_lbs = TRUE, include_date_transpos = TRUE)`. This approach is usually faster and should also make it easier to understand the returned data frame(s), especially when some variables contain missing or duplicated data. Always keep an eye on whether the `work` and `celex` columns identify rows uniquely or not.
+# execute query
+results <- elx_run_query(query)
+```
-One of the most useful things about the API is that we obtain a comprehensive list of identifiers that we can subsequently use to obtain more data relating to the document in question. While the results of the SPARQL queries are useful also for webscraping (with the `rvest` package), the function `elx_fetch_data()` enables us to fire GET requests to retrieve data on documents with known identifiers (including Cellar URI). The function for example enables downloading the title and the full text of a document in all available languages.
+One of the most useful things about the API is that we obtain a comprehensive list of identifiers that we can subsequently use to obtain more data relating to the document in question. While the results of the SPARQL queries can also be useful for web-scraping, the function `elx_fetch_data()` makes it possible to fire GET requests to retrieve data on documents with known identifiers (including Cellar URI). The function for example enables downloading the title and the full text of a document in all available languages.
## Note
-This package nor its author are in any way affiliated with the EU Publications Office. Please refer to the applicable [data reuse policies](https://eur-lex.europa.eu/content/welcome/data-reuse.html).
+This package nor its author are in any way affiliated with the EU, its institutions, offices or agencies. Please refer to the applicable [data reuse policies](https://eur-lex.europa.eu/content/welcome/data-reuse.html).
Please consider contributing to the maintenance and development of the package by reporting bugs or suggesting new features.
## Latest changes
+### eurlex 0.4.5
+
+- breaking change: `elx_run_query()` now strips URIs (except Eurovoc ones) by default and keeps only the identifier to reduce object size
+- where `elx_fetch_data()` is used to retrieve texts from an html document, it now uses by default `rvest::html_text2()` instead of `rvest::html_text()`. This is slower but more resembling of how the page renders in some cases. New argument `html_text = "text2"` controls the setting.
+- new feature: `elx_make_query(..., include_court_origin = TRUE)` retrieves the country of origin of a court case. As per Eur-Lex documentation, this is primarily intended to be the country of the national court referring a preliminary question, but other countries are present in the data as well at the moment. Recommended to interact with court procedure
+- new feature: `elx_make_query(..., include_original_language = TRUE)` retrieves the authentic language of a document, typically a court case
+
### eurlex 0.4.3
- all date variables retrieved through `elx_make_query(include_... = TRUE)` are now properly named
@@ -61,29 +72,6 @@ Please consider contributing to the maintenance and development of the package b
- fixed bug in `elx_download_xml()` parameter checking
- `elx_download_xml(notice = "object")` now retrieves metadata correctly
-### eurlex 0.4.0
-
-- download XML notices associated with Cellar URLs with `elx_download_xml()`
-- retrieve European Case Law Identifier (ECLI) with `elx_make_query(include_ecli = TRUE)`
-
-### eurlex 0.3.6
-
-- `elx_run_query()` now fails gracefully in presence of internet/server problems
-- `elx_fetch_data()` now automatically fixes urls with parentheses (e.g. "32019H1115(01)" used to fail)
-- minor fixes to vignette
-- `elx_parse_xml` no longer an exported function
-
-### eurlex 0.3.5
-
-- it is now possible to select all resource types available with `elx_make_query(resource_type = "any")`. Since there are nearly 1 million CELEX codes, use with discretion and expect long execution times
-- results can be restricted to a particular directory code with `elx_make_query(directory = "18")` (directory code "18" denotes Common Foreign and Security Policy)
-- results can be restricted to a particular sector with `elx_make_query(sector = 2)` (sector code 2 denotes EU international agreements)
-
-- new feature: request date of court case submission `elx_make_query(include_date_lodged = TRUE)`
-- new feature: request type of court procedure and outcome `elx_make_query(include_court_procedure = TRUE)`
-- new feature: request directory code of legal act `elx_make_query(include_directory = TRUE)`
-- `elx_curia_list()` has a new default parameter `parse = TRUE` which creates separate columns for `ecli`, `see_case`, `appeal` applying regular expressions on `case_info`
-
## Useful resources
Guide to CELEX numbers: https://eur-lex.europa.eu/content/tools/TableOfSectors/types_of_documents_in_eurlex.html
diff --git a/cran-comments.md b/cran-comments.md
deleted file mode 100644
index 2c74612..0000000
--- a/cran-comments.md
+++ /dev/null
@@ -1,5 +0,0 @@
-## R CMD check results
-
-0 errors | 0 warnings | 1 note
-
-* Passed R CMD Check locally
diff --git a/doc/eurlexpkg.R b/doc/eurlexpkg.R
index cbbca34..ace0486 100644
--- a/doc/eurlexpkg.R
+++ b/doc/eurlexpkg.R
@@ -108,7 +108,7 @@ dir_titles <- results[1:5,] %>% # take the first 5 directives only to save time
print(dir_titles)
-## ---- eval=FALSE--------------------------------------------------------------
+## ----dirsdata, eval=FALSE-----------------------------------------------------
# dirs <- elx_make_query(resource_type = "directive", include_date = TRUE, include_force = TRUE) %>%
# elx_run_query()
@@ -120,7 +120,7 @@ dirs %>%
ggplot(aes(x = force, y = n)) +
geom_col()
-## -----------------------------------------------------------------------------
+## ----dirforce-----------------------------------------------------------------
dirs %>%
filter(!is.na(force)) %>%
mutate(date = as.Date(date)) %>%
@@ -130,7 +130,7 @@ dirs %>%
axis.line.y = element_blank(),
axis.ticks.y = element_blank())
-## -----------------------------------------------------------------------------
+## ----dirtitles----------------------------------------------------------------
dirs_1970_title <- dirs %>%
filter(between(as.Date(date), as.Date("1970-01-01"), as.Date("1973-01-01")),
force == "true") %>%
@@ -140,12 +140,12 @@ dirs_1970_title <- dirs %>%
as_tibble()
print(dirs_1970_title)
-
## ----wordcloud, message = FALSE, warning=FALSE, error=FALSE-------------------
library(tidytext)
library(wordcloud)
+# wordcloud
dirs_1970_title %>%
select(celex,title) %>%
unnest_tokens(word, title) %>%
@@ -154,4 +154,3 @@ dirs_1970_title %>%
bind_tf_idf(word, celex, n) %>%
with(wordcloud(word, tf_idf, max.words = 40))
-
diff --git a/doc/eurlexpkg.Rmd b/doc/eurlexpkg.Rmd
index a80bb02..bf00a2f 100644
--- a/doc/eurlexpkg.Rmd
+++ b/doc/eurlexpkg.Rmd
@@ -203,7 +203,7 @@ Note that text requests are by far the most time-intensive; requesting the full
In this section I showcase a simple application of `eurlex` on making overviews of EU legislation. First, we collate data on directives.
-```{r, eval=FALSE}
+```{r dirsdata, eval=FALSE}
dirs <- elx_make_query(resource_type = "directive", include_date = TRUE, include_force = TRUE) %>%
elx_run_query()
```
@@ -221,7 +221,7 @@ dirs %>%
Directives become naturally outdated with time. It might be all the more interesting to see which older acts are thus still surviving.
-```{r}
+```{r dirforce}
dirs %>%
filter(!is.na(force)) %>%
mutate(date = as.Date(date)) %>%
@@ -234,7 +234,7 @@ dirs %>%
We want to know a bit more about some directives from the early 1970s that are still in force today. Their titles could give us a clue.
-```{r}
+```{r dirtitles}
dirs_1970_title <- dirs %>%
filter(between(as.Date(date), as.Date("1970-01-01"), as.Date("1973-01-01")),
force == "true") %>%
@@ -244,7 +244,6 @@ dirs_1970_title <- dirs %>%
as_tibble()
print(dirs_1970_title)
-
```
I will use the `tidytext` package to get a quick idea of what the legislation is about.
@@ -253,6 +252,7 @@ I will use the `tidytext` package to get a quick idea of what the legislation is
library(tidytext)
library(wordcloud)
+# wordcloud
dirs_1970_title %>%
select(celex,title) %>%
unnest_tokens(word, title) %>%
@@ -260,7 +260,6 @@ dirs_1970_title %>%
filter(!grepl("\\d", word)) %>%
bind_tf_idf(word, celex, n) %>%
with(wordcloud(word, tf_idf, max.words = 40))
-
```
I use term-frequency inverse-document frequency (tf-idf) to weight the importance of the words in the wordcloud. If we used pure frequencies, the wordcloud would largely consist of words conveying little meaning ("the", "and", ...).
diff --git a/doc/eurlexpkg.html b/doc/eurlexpkg.html
index 52575fe..c804959 100644
--- a/doc/eurlexpkg.html
+++ b/doc/eurlexpkg.html
@@ -751,7 +751,7 @@
We want to know a bit more about some directives from the early 1970s that are still in force today. Their titles could give us a clue.
dirs_1970_title <- dirs %>%
@@ -776,14 +776,15 @@ Application
library(tidytext)
library(wordcloud)
-dirs_1970_title %>%
- select(celex,title) %>%
- unnest_tokens(word, title) %>%
- count(celex, word, sort = TRUE) %>%
- filter(!grepl("\\d", word)) %>%
- bind_tf_idf(word, celex, n) %>%
- with(wordcloud(word, tf_idf, max.words = 40))
-
+# wordcloud
+dirs_1970_title %>%
+ select(celex,title) %>%
+ unnest_tokens(word, title) %>%
+ count(celex, word, sort = TRUE) %>%
+ filter(!grepl("\\d", word)) %>%
+ bind_tf_idf(word, celex, n) %>%
+ with(wordcloud(word, tf_idf, max.words = 40))
I use term-frequency inverse-document frequency (tf-idf) to weight the importance of the words in the wordcloud. If we used pure frequencies, the wordcloud would largely consist of words conveying diff --git a/docs/404.html b/docs/404.html index 0e28a81..b0c0e68 100644 --- a/docs/404.html +++ b/docs/404.html @@ -56,6 +56,9 @@
diff --git a/docs/articles/council.html b/docs/articles/council.html new file mode 100644 index 0000000..ae0af23 --- /dev/null +++ b/docs/articles/council.html @@ -0,0 +1,276 @@ + + + + + + + +council.RmdFew would disagree that the Council of the European Union (EU) – +sometimes also referred to as the Council of Ministers – is the most +important decision-maker in the EU’s legislative process. The Council +brings together the government representatives of the Member States to, +among others, negotiate EU legislation with the European Parliament as +part of the ordinary legislative procedure (OLP).
+Under the OLP, which is nowadays the most common type of law-making
+procedure, the Council should make decisions by qualified majority. In
+practice, it often decides by consensus, as Member States tend to avoid
+open disagreements. Still, enough votes are taken to give us some
+insight into the variation in Member State governments’ behaviour. We
+access these through a dedicated API maintained by the Council, which is
+also wrapped in the eurlex package.
+# packages
+library(eurlex)
+library(ggplot2)
+library(dplyr)
+#>
+#> Attaching package: 'dplyr'
+#> The following objects are masked from 'package:stats':
+#>
+#> filter, lag
+#> The following objects are masked from 'package:base':
+#>
+#> intersect, setdiff, setequal, union
+library(tidyr)
+library(stringr)
+library(ggiraph)
+
+# pull Council voting data
+cns_votes_raw <- elx_council_votes()
+
+# vote level, keep only votes with disagreements
+votes_dis <- cns_votes_raw %>%
+ select(voteProc, starts_with("countryCode")) %>%
+ select(-countryCodeNotParticipatingGrouped) %>%
+ distinct() %>%
+ filter((!is.na(countryCodeAgainstGrouped) & !is.na(countryCodeAbstainedGrouped))) %>%
+ pivot_longer(cols = starts_with("country"), names_to = "vote", values_to = "country") %>%
+ separate_rows(country, sep = "\\|") %>%
+ drop_na() %>%
+ mutate(vote = case_when(str_detect(vote, "Favour") ~ 1L,
+ str_detect(vote, "Absta") ~ 2L,
+ str_detect(vote, "Against") ~ 3L,
+ T ~ NA_integer_))
+
+# country vote counts
+country_votes_n <- votes_dis %>%
+ count(country, vote)
+
+# weighted vote proportion
+country_votes_prop <- country_votes_n %>%
+ mutate(value = case_when(vote == 1 ~ n * 1,
+ vote == 2 ~ n * -1,
+ vote == 3 ~ n * -2)) %>%
+ group_by(country) %>%
+ summarise(value = sum(value),
+ n_votes = sum(n),
+ prop = round(value / n_votes, 3)) %>%
+ ungroup()Excluding votes where all governments voted in favour, we are left
+with between 109 and 81 votes per Member
+State. While these numbers do not represent the entire historical voting
+record, they should still help us lift the veil on variation in Member
+States’ propensity to disagree. Note that due to opt-outs not all
+countries have participated in every vote.
To highlight that votes against tend to represent a strong signal of +disagreement, the following simple formula is used to calculate the +weighted proportion of Member State discontent: \(p_i = \frac{\sum_j^a{\text{infavour}_{ij}} - +\sum_k^b{\text{abstain}_{ik}} - \sum_l^c{\text{against}_{il} * 2}}{a + b ++ c}\)
+The data on votes are easy enough to plot interactively with
+ggplot2 and ggiraph. The following plot shows
+the variation in \(p_i\) where \(i\) indexes Member States.
+# viz Council votes and weighted proportion
+iplot_votes_prop <- country_votes_prop %>%
+ mutate(tooltip = str_c(country,": ", prop, ". Total number of votes: ", n_votes)) %>%
+ ggplot(aes(y = reorder(country, prop), x = prop, yend = reorder(country, prop), xend = 0, color = prop)) +
+ geom_vline(xintercept = c(0.25,0.5,0.75), color = "grey90", lty = 2) +
+ geom_point_interactive(aes(tooltip = tooltip, data_id = country),
+ show.legend = FALSE) +
+ geom_segment_interactive(aes(tooltip = tooltip, data_id = country),
+ show.legend = FALSE) +
+ theme_minimal(base_family = "Arial") +
+ theme(legend.position = "top",
+ legend.justification = "left",
+ legend.title = element_text(face = "italic"),
+ plot.background = element_rect(fill = "white", color = "grey88"),
+ axis.text = element_text(color = "grey10", size = 12),
+ title = element_text(face = "bold", size = 16),
+ panel.grid = element_line(color = "grey94"),
+ axis.title = element_text(hjust = 1, size = 14),
+ plot.subtitle = element_text(face = "italic", size = 15),
+ plot.caption = element_text(face = "italic", size = 8),
+ strip.text = element_text(hjust = 0, face = "bold")) +
+ scale_x_continuous(expand = c(0.01,0)) +
+ scale_color_gradient(low = "red", high = "navyblue") +
+ labs(x = NULL,
+ y = NULL,
+ color = NULL,
+ fill = NULL,
+ title = "Legislative discontent in the Council",
+ subtitle = "Weighted proportion of government votes in favour on contested legislation*",
+ caption = "* Only legislation with at least one vote not in favour; abstentions (x1) and votes against (x2) are subtracted from votes in favour")
+
+# interactive plot
+girafe(ggobj = iplot_votes_prop,
+ fonts = list(sans = "Arial"),
+ width_svg = 12,
+ height_svg = 8,
+ options = list(opts_sizing(rescale = TRUE),
+ opts_toolbar(saveaspng = FALSE),
+ opts_tooltip(css = "background-color:gray;color:white;font-style:italic;padding:9px;border-radius:5px;font-size:15px;",
+ use_fill = TRUE),
+ opts_hover_inv(css = "opacity:0.1;"),
+ opts_hover(css = "fill:green;"))
+)The country comparison reveals substantial variation in the frequency +of disagreement. The only Member State to ever exit the EU, the United +Kingdom, was particularly active when it comes to abstaining or voting +against legislation. On the other end of the scale is France, which has +been happy to support almost every law or Council position put in front +of it. We are unable to tell from this simple comparison whether a +supportive voting record reflects satisfaction with the negotiated +substance or governments’ overall stance on European integration (or +both).
+A more sophisticated way of scaling Member States’ preferences would +involve deriving their ideal points from the votes through an +item-response model.1 In our example, we assume that all votes +are equally informative and important; ideally, we would want to relax +this assumption.
+An example application of an ideal point IRT model can +be found in this +paper.↩︎