project:big_data_analytics [Opendata.ch Wiki]

This wiki is the archive of the Make Open Data camps and hackathons that the Opendata.ch community has run since 2011. Current events and projects are on opendata.ch, questions go to the forum.

We try to analyse bibliographical data using big data technology (flink, elasticsearch, metafacture).

Here a first sketch of what we're aiming at:

We use bibliographical metadata:

Swissbib bibliographical data https://swissbib.ch/

Catalog of all the Swiss University Libraries, the Swiss National Library, etc.
960 Libraries / 23 repositories (Bibliotheksverbunde)
ca. 30 Mio records
MARC21 XML Format
→ raw data stored in Mongo DB
→ transformed and clustered data stored in CBS (central library system)

edoc https://edoc.unibas.ch/home

Institutional Repository der Universität Basel (Dokumentenserver, Open Access Publications)
ca. 50'000 records
JSON File

crossref https://www.crossref.org/

Digital Object Identifier (DOI) Registration Agency
ca. 90 Mio records (we only use 30 Mio)
JSON scraped from API

Swissbib

Librarian:

- For prioritizing which of our holdings should be digitized most urgently, I want to know which of our holdings are nowhere else to be found.

- We would like to have a list of all the DVDs in swissbib.

- What is special about the holdings of some library/institution? Profile?

Data analyst:

- I want to get to know better my data. And be faster.

→ e.g. I want to know which records don‘t have any entry for ‚year of publication‘. I want to analyze, if these records should be sent through the merging process of CBS. Therefore I also want to know, if these records contain other ‚relevant‘ fields, defined by CBS (e.g. ISBN, etc.). To analyze the results, a visualization tool might be useful.