Beta Release v0.2

At long last we are ready to offer a v0.2 beta release of the World Historical Gazetteer (WHG) at http://dev.whgazetteer.org. We hope that spatial historians and spatio-temporal infrastructure developers will be interested in taking a look at what we are building, experimenting with their data or provided samples. It is a “sandbox,” so nothing will be saved for the time being (that will change soon). There are 5-6 months remaining in the term of our initial NEH grant, time enough to complete most of what we planned for this phase, and to incorporate more suggestions from users and potential contributors as we move toward future planning and development.

The site includes a brief guide titled “WHG Beta Release: A Tour,” which outlines what is there, what you can do and how, remaining challenges, and what is in the works. What follows is a higher level introduction.

Places and Traces

The World Historical Gazetteer is a Linked Open Data platform for publishing, linking, discovering, and visualizing contributed records of attested historical places and traces. Our initial focus has been on places, but we are working experimentally to demonstrate their integration within the platform with what we now call traces–defined as web resources about historical entities for which location in time and space is of scholarly and general interest. We are considering three classes of traces for the time being: agents (people and groups), works (e.g. artifacts, texts, datasets), and events (e.g. journeys, conflict). Our objective has been to create the first large-scale spatial infrastructure for world history: oriented toward documenting the human past at the global scale, and particularly the geography of global and transregional connections.

Our accessioning process is intended to eventually be largely self-directed; getting it to that stage means working directly and hands-on with our early contributors.

LOD Publication

Registered users of WHG can publish their place records as Linked Open Data simply by uploading them in Linked Places format (or the LP-TSV version intended for relatively simpler records). We see LOD publication as a key feature for researchers who are not in a position to stand up their own web interfaces with per-place pages. Once uploaded, each record will have a permanent URI and be accessible in our graphical interface and API; on their way to being LOD in good standing. The dataset can be browsed immediately by its owner in a searchable table and map, but turning the uploaded dataset into a contribution for accessioning requires some further steps. The data needs to have as many asserted links to name authorities as possible, and augmentations of geometry where that is missing and findable. We provide reconciliation services for that purpose.

Reconciliation

Simply put, reconciliation is the process of identifying matches between records of named entities. In this case the records are for places, and the matches are between a researcher’s records and those in existing place name authorities. So far, we provide reconciliation services for the Getty Thesaurus of Geographic Names (TGN) and Wikidata; DBpedia and GeoNames are planned.

The reconciliation process has two steps: 1) sending records to the authority, and 2) reviewing the prospective matches returned and accepting or declining them as appropriate. The results of this somewhat laborious process are 1) links, and 2) more geometry. Once augmented in this way, a dataset is ready for accessioning.

Accessioning

The last step is another reconciliation effort — this time to the WHG index. Each record is compared to the growing WHG index to determine if we have a contributed attestation for the place yet or not. If we do, the incoming record becomes a “child” or “leaf” in the set of attestations for the place. If the place is not yet accounted for, the new record becomes a “parent” — the seed for a new set of attestations. At this stage, an automatic linking can be made if two records share an authority match, but the rest will have to be reviewed as described above.

Graphical Interface

The opening screen of WHG offers users search of places and traces. We try to offer enough context on the opening screen to identify the likeliest match. Once you identify a place of interest, clicking its name take you to a “place portal” screen–where everything we have about the place, or linked to it in some way, will appear: attestations from contributors, associated traces, nearby places, physical geographic context (rivers, watersheds, ecoregions). The place portal is very much a work-in-progress at this stage. Several other features are also on our near-term to do list, including advanced search; more and better maps; user data collections; project team ‘workspace’; batching of reconciliation tasks; and more.

A Word About Architecture

There are two data stores within the WHG platform: a relational database (PostgreSQL) and a high-speed index (Elasticsearch). All uploaded data gets imported to a set of relational tables whose names correspond to the elements of Linked Places format: places, place_name, place_type, place_geom, place_link, place_when, place_related, place_description, and place_depiction. Contributed data is most readily managed in that form. Upon accessioning, records are added to the index in the manner described under Reconciliation above.

An API

This part of the WHG platform is one of the most important, and the least developed right now. Stay tuned for further developments. Our intention is to provide access to both contributors’ individual records and datasets from the database (when designated by their owner as public), and to the aggregating index records; both with numerous and useful filtering capabilities.

Content

Our index has been instantiated with records from modern gazetteer resources: 1) about 1,000 of the world’s most populous cites from GeoNames, 2) ~1.8 million place records from Getty TGN, 3) about 1,500 societies from the D-Place anthropological repository; and 4) major rivers, lakes, and mountain ranges from Natural Earth and Wildlife Research Institute.

To this modern “core” we have begun adding historical data: 1) 10,600 entities harvested from the index of the Atlas of World History (Dorling Kindersley, 1995), offering broad but shallow global coverage; and 2) our first specialist gazetteer, HGIS de las Indias, which consists of approximately 15,000 settlements and territories in colonial Latin America. There are several additional large datasets in the queue, which we will be adding in partnership with contributors. Some are previewed as heat maps on our Maps page.

Broad coverage of modern names with increasing historical depth and connections supplied by trace data.

Our Pelagios Connections

The WHG platform borrows extensively from the Peripleo application developed by Rainer Simon of the Pelagios project, extending it significantly in a few ways. Our backend architecture closely mimics that underlying both Peripleo and the Recogito annotation tool, and we are actively collaborating with Rainer and the entire Pelagios Network team on several aspects of this work. In particular, we are co-developing the data format standards for contributions to both systems: Linked Places format, and a nascent Linked Traces annotation format.

Feedback

We welcome suggestions, critiques, even praise :^) – and there is an email form on the site which makes it easy to offer it. Please bear with us in this active development stage and check back as we realize the system’s potential more fully over the next several months. Look for further blog posts and follow us on Twitter; we tweet progress and related information as @WHGazetteer and @kgeographer.

 

Linked Traces progress

As described in our last post, the World-Historical Gazetteer (WHG) and Pelagios projects have adopted the term “trace” to refer to historical entities for which there is spatial-temporal data of interest, including events, people, works, and other artifacts. Following the lead of Pelagios’ Peripleo, the WHG system (initial beta release July 2019) will index contributed trace data, linking them to places in an underlying knowledge graph that is a) navigable in graphical features, and b) queryable in an API.

We (WHG and Pelagios) have set out to create a standard Linked Traces data format (LTAnno), which will take the form of W3C web annotations. We welcome (need, actually) active collaboration in that modeling task, and feedback from interested observers.

An LTAnno target is an LOD web-published record of some entity, and its body contains a) a place record URI, b) a relation between the place and the target entity, and c) an optional temporal scoping for each. It should be possible to have multiple bodies per target (per example below) and multiple targets per body (e.g several people having the same birthplace, several works having the same place as subject, and so on).

A Trace Data Example

WHG will fully support LTanno format, and likely focus on a few types of trace data, including those related to geographic movement such as journeys and cultural diffusion. Figures 1 and 2 illustrate a test Journey record using the draft LTAnno format. In it, 38 annotation bodies referring to WHG place URIs are linked to a single target, the WorldCat record for the source of the waypoints, “Xuanzang: a Buddhist pilgrim on the Silk Road” (Wriggins, 1997). A user finding their way to any of the 38 places will learn they were waypoints on the journey, be able to see the others, and to navigate to their respective place pages. This only scratches the surface of what will be possible, given a growing volume of trace data for other events, people, and works.

Figure 1 – Portal page for Bamiyan in World-Historical Gazetteer (in development)
Figure 2 – Each waypoint for the Journey trace is linked to its own place portal

Next Steps

The draft examples of LTAnno for different trace types are only preliminary, for discussion. In the coming weeks, Rainer Simon and I will coordinate development of a spec our respective projects can support. There is a Google Group and email list for this working geoup, and we will go into further detail about the draft spec there shortly. Active collaboration by data modelers, data providers, and future users of the format is most welcome.

One of the first orders of business is gathering a few sample datasets for different types of traces, in order to better understand the variety of modeling circumstances. These will then have to be converted into early versions of the format to test usability and usefulness.

At a later stage, we’ll have to put together a simple Linked Pasts ontology describing terms introduced by both this new LTAnno format and the recently developed Linked Places format for gazetteer data connectivity.

Linked Traces

Linked Pasts is an annual symposium. Linked.Art and Linked Places are data models with associated format specifications. Can we manage one more Linked something? Rainer Simon and I have begun an initiative to develop a Linked Traces annotation model and file format as a standard for contributions to linked open data aggregation projects such as World-Historical Gazetteer and Pelagios. The effort could easily extend to software and systems for displaying, searching, and analyzing trace data. The idea has drawn considerable interest, so here are some thoughts to start a discussion…and action.

What is a trace?

For our purposes a trace is any historical entity having a spatial-temporal setting (i.e. footprint) of interest—very general! The types of traces we’re immediately focused on include: people and groups of people, events of any complexity, and artifacts of all kinds (e.g. objects, texts, art works).

What is trace data?

Trace data are annotations of web-published records about (and images of) trace entities. We posit here that the body of a trace annotation must include a place reference (URI and name/title), should include a relation (e.g. waypoint, findspot, birthplace), and could include a temporal scope for that relation. Properties like creator and date are musts also. Trace data should take the form spelled out in the W3C Web Annotation Model and Vocabulary, in the JSON-LD syntax of RDF. Draft examples of some trace annotations have been posted in a GitHub repository for discussion. There are a few outstanding issues that need community consensus to resolve, outlined below.

Why trace data?

The Peripleo pilot application launched a few years ago by the Pelagios project is an example of traces in action. Underlying Peripleo is an index of a) place records aggregated from multiple gazetteers, and b) what we are now calling trace data: annotations of records about ancient coins, coin hoards, and inscriptions with relevant locations such as find spots.

There are many other kinds of things associated with places—at times or during periods—we might like to see, compare, and analyze as elements of “deep” linked data place records in future Peripleo-like software (e.g. World-Historical Gazetteer, now in development). For a given place, discover not only what museum artifacts or inscriptions were found there, but what historical persons are associated with the place, and in what way; what journeys of exploration or pilgrimage it was a waypoint on; and what texts and art works it is a subject of.

We have already heard from people with Person and Event data, and Rainer notes that this should support annotations of IIIF-formatted manuscripts and other images.

One sample

Here is one sample draft annotation record for a Journey event. As mentioned, more examples are on GitHub.

{ "@context":[
    "http://www.w3.org/ns/anno.jsonld",
    { "lpo": "http://linkedpasts.org/ontology/lpo.jsonld"}
  ],
  "id": "http://my.org/annotations/92837",
  "type": "Annotation",
  "creator": {
    "id":"http://example.org/people/2345",
    "name":"Ima Tracemaker",
    "homepage":"http://tracemaker.org"},
  "created": "2019-03-18",
  "motivation": "linking",
  "body": [
    {"id": "http://whgazetteer.org/places/86880",
     "dc:title": "Tashkent",
     "lpo:relation": "lpo:waypoint",
     "lpo:when": {"timespans":[
       {"start":{"in":"630"},"end":{"in":"630"}}]}
    },
    {"id": "http://whgazetteer.org/places/84774",
     "dc:title": "Mathura",
     "lpo:relation": "lpo:waypoint",
     "lpo:when": {"timespans":[
        {"start":{"in":"634"},"end":{"in":"634"}}]}
    },
   // ... etc.
 ],
 "target": {
   "id": "http://my.org/events/90001",
   "type": "lpo:Journey",
   "dc:title" "Pilgrimage of Xuanzang"
 }
}

Open questions

The next step is for a working group to collectively answer existing open questions, and to surface (and answer) questions we haven’t thought of. We welcome collaborators and observers. A few questions that came to mind while developing the prospective samples:

  1. What are the types of traces (annotation targets)?
  2. Should there be a vocabulary of type-specific relations? E.g. waypoint for Journey traces, or birthplace for Persons.
  3. How can bodies (place/time assertions) be combined as sequences in sets for a given target? E.g. Journey waypoints.
  4. How can relations be combined in sets? E.g. a Place was both a birthplace and deathplace for a Person.
  5. Where can “when” be expressed in an annotation? E.g. in #4, can a date be associated with each relation to the Place?
  6. How should “extension” terms we introduce (and allowed by the W3C spec) be defined? In a “Linked Pasts Ontology”? What will be its contents?

Undoubtedly more will surface.

Next steps

I’ve created a Google Group email list for tracking conversation amongst collaborators and observers, and posted parts of this document as an editable Google Doc. After some initial feedback perhaps we should have a Google Hangout. (My plan to begin extracting Google from my life is not going well!) Suggestions for other tools and platforms are welcome.

 

 

Progress Report, Dec 2018

Data pending accessioning to WHGazetteer, Dec. 2018 (partial)

The World-Historical Gazetteer project (WHG) is now just past the halfway mark in our three-year NEH grant period. We are on track to produce what was promised in the proposal, notwithstanding some expansion of scope and unforeseen developments and opportunities. We expect future project milestones roughly as follows:

  • March 2019: Alpha release of functional contributions system and basic API
  • July 2019: Limited public beta for comment: contributions, API, place pages
  • July 2019: Second Advisory Committee meeting
  • September – December 2019: Rolling beta releases and data additions
  • 1 April 2020: Launch
  • April – May 2020: Refinements

There are currently seven (!) activity tracks: 1) data development; 2) outreach to potential data partners; 3) format standards; 4) backend design and development (data stores and contribution pipeline); 5) frontend design and development; 6) API; and 7) ecosystem.

The primary focus right now is the contribution pipeline of #4 and some pilot screens below show progress. We have a considerable volume of data queued up for conversion to Linked Places format then accessioning, and much more is promised.

The functionality we are providing for contributing data has become more ambitious than originally envisioned. We need to accommodate contributions ranging in size from a handful to many thousands, with a minimum of hand-holding by WHG support staff.

Registered users will have a dashboard from which they can manage datasets through the entire workflow: (i) uploading files, (ii) ingesting to our database, (iii) initiating reconciliation against name authorities, (iv) reviewing and validating “hits”, and (v) submitting the resulting enhanced dataset for indexing. Users will also be able to create, download, and share “MyPlaces” collections of records tagged while using the site. That’s a lot!

I will be presenting this poster at the Mainz Linked Pasts IV meeting, 11-14 Dec. The following figures are excerpted from it.

I will also join presenters of the LatAm Gazetteer project funded by a Pelagios Commons Resource Development grant. The LatAm project has provided us two excellent datasets, which together will form within WHGazetteer a “seed” of sorts for Colonial Latin America. Progress working with that data is indicated on the following functional screens in our pilot (pre-alpha) site. Note that little to no attention has been paid to styling at this point.

Upload file to begin dataset creation
List and manage datasets in progress
Review results of reconciliation to Getty TGN, marking correct “hits”

Contributing in Linked Places format

HGIS de lasIndias spatial footprint
Fig. 1 – HGIS de las Indias spatial footprint

We are pleased to announce version 1 of the Linked Places (LP) format, to be used for contributions to World-Historical Gazetteer (WHG) and to the Pelagios “registry index” that underlies its Recogito annotation tool and Peripleo search interface. We were joined in this effort by Rainer Simon of Pelagios and Graham Klyne and Arno Bosse of Oxford’s Cultures of Knowledge, so a big shout of thanks to all three.

A basic specification of the format has been published to GitHub, along with some relevant files:

The spatial footprint of the HGIS de las Indias contribution appears in Figure 1; more about that data and ongoing efforts to make it the “seed” for a virtual LatAm Gazetteer within WHG are forthcoming soon.

An Interconnection Format, Not “The OneModel

Historical research projects producing gazetteer data have distinctive data models reflecting their source data and project-specific requirements. We are completely agnostic as to contributors’ internal models and formats. The Linked Places format provides a uniform way to build links between different gazetteers.

Temporal Scoping

WHG and Pelagios are aggregating contributions which have great variety in scope and granularity. With LP format, we are striking a balance between accommodating detail and offering simplicity. Two underlying conceptual models: of Place and of attestations. We say (hopefully not controversially) that places have, over time,

  • one or more names in various languages
  • one or more functional types
  • zero or more known spatial footprints (geometries)
  • one or more relations to other places

Gazetteer data developers gather attestations of these properties, often temporally scoped. That is, names, types, geometries, and relations were/are true for some timespan—whether we have that detail to hand or not. Thing is, some projects do and some don’t, and we must accommodate both cases.

The simple subject, predicate, object form of RDF is not adequate for these kinds of relations and so we have reified them as attestations: NameAttestation, TypeAttestation, Setting, and RelAttestation. Each can be temporally scoped with a “when” element and include citation information, although this is not required. We also permit (and encourage) a global “when” element for the record—effectively the union of temporal attributes for names, types, geometries, and relations.

Links

We ask that all contributions include, if at all possible, at least one closeMatch or exactMatch relation to a published place record having a de-referenceable URI. These links are “coin of the realm” so to speak. If an incoming record shares such a link with an existing record in our registry index, we  add it to the existing record. If it shares no such link with any existing records and doesn’t otherwise match name, type, and geometry, a new registry index record will be created using it as a seed.

Three other kinds of LinkAttestations to existing web resources are recognized: subjectOf, primaryTopicOf, and seeAlso.

Record-Level Properties

Several “record-level” properties round out this format, as detailed in the GitHub specification: title, ccode (modern country), description(s), and depiction(s).

Moving Forward

Please get in touch  if you are interested in contributing to WHG. The Linked Places format is not set in stone; this v1 is subject to revision based on our experiences ingesting contributions. The ingest of HGIS de las Indias data as a gazetteer offered to users of Recogito has succeeded, following significant manual effort to make the transformation from that project’s complex model to LP format. Temporally scoped namings and parthood relations in that dataset made it over, but are not reflected in Recogito. Simpler example datasets will be added soon. The WHG web interface and API are being designed to expose all contributed place attributes, but they won’t be available for several months.

Linked Places Annotations

Next up in our format-creation effort is an update to the standard format for contributions of annotations. Just as Peripleo displays coins and inscriptions associated with places in its registry index, the WHG graphical search will such annotation records, but in our case for historical routes, datasets, and bibliographic records. The format previously used by Pelagios/Peripleo will be updated soon—details to follow.

Contributing to World-Historical Gazetteer: a Preview

The World-Historical Gazetteer project (WHG) will soon begin aggregating and indexing historical gazetteer datasets, and exposing them as Linked Open Data via graphical and programmatic web interfaces — just as Pelagios Commons’ Peripleo project has done for a few years. And like Peripleo, WHG will also index contributions of annotation records that associate historical “items” with place identifiers. Typical items for Peripleo have included coins, coin hoards, and inscriptions of the Classical Mediterranean. Items records WHG will focus on include journey events, regions, and datasets. In fact, annotated items could be anything for which location is relevant, e.g. people and various types of events.

We are almost ready to begin accepting contributions; this post previews the pipeline and formats involved.

Contributions to WHG can include, in some combination: 1) gazetteer data, i.e. place records drawn from historical sources; 2) annotation records that associate a published record about an item with a place identifier; 3) collections of item metadata records referenced in annotations; and 4) a file describing the contributed dataset(s) in Vocabulary of Interlinked Datasets” format (VoID).

Over the past several weeks we have collaboratively developed a new Linked Places format (LPF here for short) with Rainer Simon of Pelagios, to be used for contributions of historical place data to both WHG and Pelagios’ Peripleo. The Linked Places format is designed around the JSON-LD syntax of RDF (it is also valid GeoJSON, with temporal extensions, as explained in the GitHub README). The new format makes use of several existing vocabularies and also introduces some terms specific to our shared purposes.

Several expert colleagues contributed valuable input, including Graham Klyne, Richard Light, Lex Berman, Arno Bosse, and Rob Sanderson [1]. We are in the process of updating the template Peripleo has used for annotation contributions (formerly Open Annotation in RDF Turtle, now its next-generation W3C Web Annotation in JSON-LD). Both are discussed in a little more detail below.

Contributing historical place records

There will be two separate workflows for contributions: from larger projects and from smaller ones. The distinction is whether a project has the capability and resources to meet two criteria which are accepted norms for publishing Linked Data: 1) publishing data in some syntax of RDF (in our case the new LPF); and 2) providing a unique URI and associated “landing page” for each resource described.

Case 1: Larger projects

If your project has (or will have) a web presence that provides public pages describing your individual places and/or “items”, (routes, regions, etc.), then we ask that you perform a transformation and export of your data in the standard formats mentioned above – Linked Places, a future annotation format (see Contributing Annotations below), and VoIF. Upon validation, we will ingest those records, link them with those already in the system, and expose them in a nice GUI and API. Details of WHG interfaces are forthcoming soon.

Case 2: Smaller projects

If your project does not entail creating a web site providing per-record landing pages, then we can accept your data contribution as CSV, mint unique URIs, and provide very basic landing pages for places and other items. The records will also be made available as JSON-LD (bonafide RDF) via our API. We will provide a Python program for converting CSV to LPF, but note that the CSV will have to conform to a template that aligns with LPF (available soon). Conversion from your native format to our CSV template will probably be more manageable than to LPF. In other words, upon submitting CSV data we can parse, a semi-automated conversion and ingest procedure will result their publication as Linked Open Data.

Contributing annotations

WHG will index metadata describing historical “items” annotated with gazetteer record identifiers. These annotation records assert, in effect: “this item is/was associated with this place, in this way;” and optionally, “at this time.”

The result of such annotations can be seen in the current Peripleo interface, where upon navigating to a given place, you can view metadata (including images) for coins and inscriptions associated with it in e.g. a foundAt or hasLocation relation. Annotations exposed in the WHG web interface will include historical journeys for which the given place was a waypoint, and regions, works, and datasets including or referring to the place.

Annotation contributions will comprise two sets of data: 1) collections of brief Item metadata records; and 2) collections of annotation records in W3C Web Annotation format. The contribution template in use by Pelagios’ Peripleo now is currently being updated to better account for typing of items and relations. Details of that new Linked Places annotation format (LPAF?) will be published soon. Collaborators in that modeling effort are most welcome!

[1] Twitter handles, in order: @gklyne, @RichardOfSussex, @mlex, @kintopp, and @azaroth42