Overcoming challenges in DiScholEd’s development: a journey of problem-solving and design enhancements

As one of the interns involved in the development of the application, I encountered several challenges throughout the process. This blog post outlines the problems faced, the attempted solutions, and the ultimate resolutions achieved, along with additional design enhancements. Let’s delve into each issue and its respective solution.

Unexpected text display in the index 

The initial hurdle involved the index appearing even when there were no named entities in the text. Furthermore, the index displayed the entire text instead of the intended entries. After thorough investigation, it was revealed that the problematic indexes were generated using the <div class="tei-p17 indent"> tag, which lacked proper accompanying tags.

The first approach : trying to implement JavaScript

To address the issue, an attempt was made to utilize JavaScript by adding a <script> tag to the HTML template. The intention was to remove the problematic <div class="tei-p17 indent"> tag and replace it with appropriate titles. However, this approach proved ineffective due to the dynamic generation of the HTML that used the Document Object Model (DOM).

Understanding the DOM

The DOM provides a programming interface that allows developers to interact with and modify the content, structure, and style of an HTML document. It enables scripting languages like JavaScript to access and manipulate elements and properties within the document. When a web page loads in a browser, the DOM creates a representation of the HTML document, enabling dynamic updates and interactivity through scripts.

Challenges with dynamic generation

When building a website, the structure of the page is defined using HTML, and this structure is represented by the HTML Document Object Model (DOM). To manage certain specialized tags like <pb-view> or <pb-browse>, a JavaScript bundle is used. When developing web applications, developers often use different JavaScript libraries, frameworks, and modules to add functionality and interactivity to their websites. Each of these components typically consists of multiple JavaScript files. However, loading numerous separate JavaScript files can lead to increased network requests and slower page loading times, affecting the overall user experience. To mitigate these issues, JavaScript bundling is employed. In my specific case, I had a <pb-browse> tag in my HTML file. When the application loads, it first processes my HTML, applies my script along with the <pb-browse> tag, and then waits for the page to finish loading. Once the page is loaded, the JavaScript bundle kicks in and generates all the titles, text, and other elements based on the structure defined by the tags. However, this caused an issue because the script was designed to work with a website that already had content, not an empty one. Even if I tried applying the script after the site was initially created, its effects would only be visible when using a specific feature or refreshing the page.

Seeking assistance

Considering the challenges faced, seeking guidance from the community seemed like a logical step. Consulting the Slack community , I inquired about executing the script whenever changes were detected within the index’s <div>. However, concerns were raised regarding the script’s viability, as it relied on specific tags that could have been modified during the bundle JavaScript upgrade.

Deeper investigation

Analyzing the ODD, following a recommendation from a Slack community member, I conducted a thorough examination to identify the underlying problem. Through careful analysis and comparisons between functioning and affected pages, I discovered an error in the rule for the <p> tag, which required modification to the “omit” setting. This adjustment successfully resolved the first issue where the text would be duplicated in the index section even if there were no named entities in the text.

Missing <p> tags 

While the previous problem was resolved, I still had to resolve the presence of the text and index within the same section. Closer examination revealed that certain pages lacked <p> tags as they were part of a larger paragraph encompassing the entire page. To address this,  and after discussing it with my tutor Floriane, it was decided to add closing and opening <p> tags accompanied by justifying comments whenever such deficiencies were encountered in the file.

Identifying affected files

The next step involved identifying all files lacking <p> tags within their <pb> sections. For this purpose, a Python script was created to scan the files, record their filenames, and note the corresponding <pb> tags where <p> tags were missing. A .csv file was generated to track the problematic files. This script successfully executed on 714 files, revealing that 108 of them had the identified issue.

Resolution using regex

To rectify the problem, a regular expression (regex) was employed to add the missing <p> tags at the appropriate locations within the affected files. This systematic approach effectively resolved the challenge, ensuring correct display of the text and index.

Appropriation of the Pb-Timeline functionality

How it works

A <pb-timeline> is a special tag from the bundle, it displays a timeline for all the files in a collection. To implement the pb-timeline feature in the platform, a custom API rule needed to be created. The API was designed to return a JSON object, where each property represented a date or the special marker “?,” indicating undated resources. Each entry contained a value corresponding to a count of resources or an object with “count” and “info” properties. The “info” property comprises an array containing HTML code to be displayed as a list within the tooltips.

Here is an example of the expected JSON with the good format:

{
  "1766-11-21": {
    "count": 1,
    "info": [
      "<a href=\"/exist/apps/discholed/bi/corpus/Brief03vonJAEuleranJHSFormey.xml\" part=\"tooltip-link\">Letter from John Albert Euler to Jean Henri Samuel Formey (Saint Petersburg, 21 novembre 1766)</a>"
    ]
  },
  "1766-12-25": {
    "count": 1,
    "info": [
      "<a href=\"/exist/apps/discholed/bi/corpus/Brief04vonJAEuleranJHSFormey.xml\" part=\"tooltip-link\">Letter from John Albert Euler to Jean Henri Samuel Formey (Saint Petersburg, 25 décembre 1766)</a>"
    ]
  },
  "1766-08-15": {
    "count": 1,
    "info": [
      "<a href=\"/exist/apps/discholed/bi/corpus/Brief02vonJAEuleranJHSFormey.xml\" part=\"tooltip-link\">Letter from John Albert Euler to Jean Henri Samuel Formey (Saint Petersburg, 15 August 1766)</a>"
    ]
  },
  "?": {
    "count": 2,
    "info": [
      "<a href=\"/exist/apps/discholed/bi/corpus/Boeckh_Buchkatalog.xml\" part=\"tooltip-link\">Catalogue of my books</a>",
      "<a href=\"/exist/apps/discholed/bi/corpus/Boeckh_Plan.xml\" part=\"tooltip-link\">Plan for arranging the philological seminar at the university of Berlin (copy)</a>"
    ]
  }
}

While other applications used an XQuery rule to generate the timeline, we opted, with Floriane, to use a Python script to produce an XML file. This XML file was then transformed into the expected JSON format when the API was called.

Bundle JavaScript 

Although the timeline was successfully displaying on the site, we encountered an issue with the entries inside each container. The pb-timeline did not show the list of documents when the number of entries exceeded ten. To address this, modifications were required in the bundle JavaScript, but direct access to the generated bundle URL was not available. Nevertheless, local changes were made, and exploration for deploying these modifications on the server is still underway. Additionally, a unique HTML code was created for each collection, enabling the implementation of a distinct timeline for each collection.

Index of places rework and corrections 

The original index of places generated this HTML page

Let’s begin by focusing on the design. The current page displays an overwhelming amount of information, making it challenging to find specific details easily. Additionally, when hovering over a place, the HTML <div> shifts and centers the information on the screen. The page exhibits excessive movement while the map remains static and underutilized. There are also some bugs, such as the map not responding correctly when hovering over a location.

Corrections of the ODD

Before delving into the design, I needed to fix and modify how the page is displayed using HTML and ODD. My primary objective was to create categorized lists of places based on their attributes, specifically “@type”.

To accomplish this, I devised a block model within the ODD for each possible @type. As a starting point, I developed a solution for one @type. The ODD implementation involves creating a list using the <pb-collapse> element, which is ideal for displaying lists. Within this structure, a title is created for each category, such as “Building.” In the content section, the ODD fetches and alphabetically orders every place with the attribute @type=”building”. Each place name is then transformed into a <pb-highlight> and a <pb-geolocation> element, enabling the map to move to the desired location when the place name is hovered over.

TEI Publisher’s channel communication

The movement of the map is facilitated by utilizing the <pb-geolocation> tag, along with the “emit” and “subscribe” attributes. With TEI publisher, it is possible to emit information on a channel (e.g., emit=”map” sends information to a “map” channel created by the application) and subscribe to specific channels. When hovering over a place name on our page, the <pb-geolocation> activates and sends the coordinates via the “map” channel. Simultaneously, the <pb-leaflet-map> component (used for displaying the map) listens to the “map” channel, receives the coordinates, and adjusts the map view accordingly.

Once I successfully implemented the above functionality for one collection and one attribute (@type=’building’), I proceeded to develop a Python script to extract all possible @type attributes from the index of places. Furthermore, I added a block in the ODD to handle places that don’t have a @type, as some collections utilized different attributes such as @xml:id. 

The Python script yielded the following list of possible @type attributes: 

['country', 'region', 'city', 'plain', 'river', 'continent', 'sea', 'castle', 'village', 'island', 'peninsula', 'lake', 'valley', 'mountain', 'pass', 'castle', 'fort', 'monastery', 'state', 'mansion', 'building', 'waterfall', 'town', 'strait', 'district', 'municipality', 'hamlet', 'house', 'street', 'square', 'archeological_site', 'department', 'embassy', 'stream', 'camp', 'canton', 'park', 'canal', 'theater', 'cathedral', 'thoroughfare', 'empire', 'forest', 'gulf', 'hotel', 'town hall', 'country', 'steam', 'ocean', 'monument', 'quarter', 'point', 'port', 'church', 'university', 'boulevard', 'station', 'place', 'hill', 'opera', 'college', 'circus', 'venue', 'cabaret', 'duchy', 'museum', 'upland', 'settlement', 'location', 'mountains']

Subsequently, I used another Python script to generate a block for each attribute in the ODD. Following a comprehensive review of every index, it became apparent that the Berlin Intellectuals collection was not functioning correctly with the ODD. Upon further investigation, I discovered redundant specifications in the ODD that could cause errors in the presence of duplicate entries in the index file. Consequently, I made necessary modifications to the ODD and notified Floriane about the duplicates.

 

Minor design modification

Designing collection and files display

Another aspect of my work involved improving the design of the application, particularly the collections and their display. To enhance user experience, I modified the HTML code, making all the blocks clickable, not just the titles. Additionally, the display of the collections was modified from a single column to two columns, resulting in a more visually appealing representation.

Upgrading splash backgrounds

For each collection, I implemented a splash background which is a loading image used as a background. When loading a page, a corresponding splash is called and displayed while the page is loaded. At the beginning of my internship, the application did not have a splash implemented. Consequently, when loading the page, multiple random information, such as the available languages in the application, would be displayed instead of a designated splash screen. As part of this enhancement, the decision was made to update all the images in the collection to higher quality versions, further enriching the user experience.

Refining metadata display

Addressing a minor issue with metadata, I upgraded the ODD by creating a rule for each entry. This ensured that when the information was lacking, the entry would not be displayed, ensuring that no useless information was present. While working on this task, I also identified an opportunity to upgrade the CSS of the metadata facets, enhancing the overall visual appeal and readability.

Conclusion

Throughout the development process, I tackled various challenges related to index display, missing tags, and design elements. By implementing appropriate solutions, seeking community guidance, and conducting in-depth investigations, I successfully resolved these issues and enhanced the functionality of the application. The modifications to the pb-Timeline, collection display, splash backgrounds, and metadata facets have resulted in an improved user experience and a more visually pleasing application design.

---

---