Design Thinking and Knowledge Engineering : A Machine Learning Case

Knowledge and design captured in the heads of people is a starting point for the abductive design process in which design thinking and its tools support the ideation and externalization of design prescriptions. Turning these design prescriptions into innovative solutions implies to further decompose design thinking artifacts while enriching them with semantics, as this enables a bridging opportunity to conceptual models representing actual enterprise and operational contexts. Realizing the bridging opportunity requires to intertwine digital twins of design thinking artifacts with the semantic expressiveness of conceptual modelling methods. The problem is that this requires a considerable knowledge engineering effort. To ease the problem, machine learning concepts are introduced into the Design2Model approach employed in OMiLAB. Furthermore, a corresponding tool is realized. As a result, the paper at hand reports on a novel application case in which machine learning and knowledge engineering is combined to realize – in an effective way – the bridging opportunity between design thinking and conceptual modelling. In particular, the case covers a component of the Design2Model tool responsible for digitalizing, semantifying, and processing of Post-it Notes from a design thinking workshop and their representations in diagrammatic conceptual models. The report focuses on technologies used and their qualitative artificial ex-post evaluation in a laboratory environment.


I. INTRODUCTION
The digital transformation age imposes the challenge on enterprises to innovate their service offerings as disruptive changes in society and new technological inventions impact existing business models. Design thinking is a possible vehicle to understand the innovation problem before translating the resulting business insights into practical solutions [1]. Knowledge management frameworks support this translation [2]. Corresponding to these frameworks, the exploitation of innovative services based on technological inventions like proposed in [3] is subjects to needs and constraints from society and marketplace ecosystems. Accordingly, two approaches can be identified that have to be bridged for service innovation. The first is to create business insights through design thinking in a design environment. The second is to realize technological inventions through engineering in an operation environment. Conceptual modelling is proposed as a glue between the two approaches featuring two levels of interpretability [4]: one for people engaged in design thinking and one for the operation of technological systems.
While the strengths and benefits of conceptual modelling are independently known for design and operation environmentse.g., first progress has been made on applying conceptual modelling in design environments to help facilitate business insights through design thinking [5], and conceptual modelling has been successfully applied in engineering to facilitate inventions in operation environments (see, e.g., [6])the idea of employing conceptual modelling as a glue between the two approaches has only been recently formulated in a domain-independent framework provided by OMiLAB (www.omilab.org) [7]. The overall goal of this framework is to enable a highly descriptive (modelling) environment for the communication of everything to everything, e.g., service design and industry 4.0.
The part of this framework that this study focuses on is called the Design2Model approach which is illustrated in Fig.  1. The aim is to make further progress on connecting design thinking and conceptual modelling, which is motivated by the idea that this connection offers benefits for the design thinking and conceptual modelling communities. On the one hand, participants in design thinking can benefit from the automated processing capabilities of conceptual models enabling access to mechanisms and algorithms for, e.g., intelligent design exploration, large-scale crowdsourcing of designs on the web, and semi-automated operationalization of design. On the other hand, conceptual modelling benefits from design thinking as it provides an alternative to traditional knowledge engineering approaches employed for building knowledge bases. In detail, the Design2Model approach is concerned with engineering digital twins of design thinking artifacts and their semantic lifting to match the expressiveness of conceptual modelling methods. This involves typical knowledge engineering tasks like acquiring knowledge and design captured in the heads of people and a further transformation into semi-formal representations that can be understood by people but that can also be processed (semi-)automatically by computer systems. A high effort is needed for these knowledge engineering tasks as design thinking requires great agility and the conceptual model-based digital twins of design thinking artifacts must reflect this agility requirement. The high knowledge engineering effort is a problem as it introduces costs, complexity, and other risks. To ease the problem, this paper explores how to combine knowledge engineering and machine learning. This exploration takes the form of tooling support that is engineered for the Design2Model approach. Likewise, the method employed in this paper is to report on a component of the Design2Model tool that realizes a combination of machine learning and knowledge engineering to connect design thinking and conceptual modelling. In particular, the component takes as an input a (live) recording of a Post-it Collaboration Workshop which is a design thinking setting and method useful for creating business insights [8]. The output of the component is a digital and semantically enriched conceptual model of the workshop results supporting further semi-automated processing. Realizing this component requires a combination of (1) machine learning used to enable the recognition of Post-it Notes from a recording and for handwritten text recognition and (2) knowledge engineering used to conceptualize, implement, and deploy knowledge structures and further processing capabilities based there upon. Thereby, the focus in on applying existing foundations for a new application case.
The paper is organized as follows. The concept, state of the art, and architecture of the Design2Model tool component is covered in Section II. In Section III, the technical approach for realizing the component is described. Experimental results can be found in Section IV. A discussion and a conclusion are provided in Section V and VI.

II. CONCEPT, RELATED WORK, AND ARCHITECTURE
OMiLAB provides a framework to engineer digital products and to facilitate service innovation by bringing together business strategists, decision makers, and designers with engineers, business execution, and intelligent environments. This requires bridging two different challenges. On the one hand, the challenge is to deal with a high variability at design-time, which is a result of ideating hypothetical application scenarios. On the other hand, the challenge is to deal with complex dynamics at run-time, which is a result of executing behavior in the real world. To bridge the gap, conceptual models are considered as first-class citizens, as they are the binding glue between the decomposition of design time requirements (e.g., from design thinking) and the abstraction from run-time specifications (e.g., cyber-physical systems) [4].
The result of employing conceptual models for bridging is the three-layer framework shown in Fig. 2. Infrastructure and software are the hidden layers that enables efficient knowledge engineering in this framework. However, OMiLAB not only provides a framework, but rather understands this framework as an invariant among others to build a network of physical and virtual laboratories. Such laboratories are realized in different OMiLAB Nodes worldwide [7]. In these laboratories, workshops are held to facilitate digital products and innovative service offerings for different domains as a reaction to disruptions in the digital transformation age. Focusing on tackling design time challenges and on decomposing the result thereof is the Design2Model approach. This approach plays a crucial role in the framework provided by OMiLAB and is employed in workshops conducted in the laboratories of the OMiLAB network and elsewhere. Current tool support for the Design2Model approach is using a vision-based system to digitalize design thinking artifacts originating from the Storytelling with Scenes design thinking setting and method [6]. This system shown in Fig. 3 is based on the detection of markers attached to haptic elements used in design thinking, inspired by the work of [10]. Each marker contains a binary matrix determining a marker's identity. The semantic lifting of detected markers is performed based on a domain-specific ontology that maps a marker's identity to its semantics. ADOxx has been used according to [11] to develop a conceptual modelling tool for importing the identified and semantically enriched markers in form of a diagrammatic modelling language representing the design thinking artifacts. The result is a conceptual model-based digital twin of the design thinking artifacts. Further decomposition and processing capabilities are provided in the tool as well. Defining a set of figures with corresponding markers and providing a domain-specific ontology for semantic enrichment requires a high knowledge engineering effort. Knowledge about appropriate figures must be gathered on a project-level basis. Additionally, the semantics of each figure must be engineered in an ontology and linked to the semantics of diagrammatic modelling elements.
Rather, this paper focusses on utilizing machine learning to recognize the elements used in design thinking. The idea is that this allows for more flexibility and less setup time for design thinking workshops compared to tag-based technologies. The class of machine learning algorithms selected is called semantic image segmentation. While semantic image segmentation can be applied in the Storytelling with Scenes design thinking setting and method to recognize the figures used, it also enables support for additional design thinking settings and methods, e.g., a Post-it Collaboration Workshop, thus extending the scope of the Design2Model tool. In the latter case, machine learning is not only required to recognize Post-it Notes through semantic image segmentation but machine learning is also employed to recognize handwritten text on Post-it Notes. A 100%-service based approach is employed to develop the new component of the Design2Model tool for Post-it Collaboration Workshops. Six kinds of services make up the component under scrutiny. The first kind of service called Multimedia I is responsible for managing video recordings of the vision-based system. Furthermore, this service allows the selection and management of frames to be further processed and provides means for pre-processing of images according to the requirements of the other services. The second kind of service called Image Segmentation is responsible for creating masks that classify images into semantic regions. With proper training, the masks can be used to extract from an image the elements employed in design thinking, e.g., the Post-it Notes on a whiteboard. The third kind of service called Multimedia II is responsible for extracting the masked areas from images, for identifying areas of handwritten text on the extracted areas, and for performing normalization and pre-processing actions. The fourth kind of service called Text Recognition is responsible for recognizing handwritten text. The fifth kind of service called Digital Design View is responsible for importing the results of the previous ones into a diagrammatic conceptual model and for further processing, e.g., semantic annotation, RDF export and validation, and ontology-based reasoning. The sixth kind of service called Service Orchestration is responsible for integrating and executing the previous ones. Fig. 4 shows a mock-up of the sixth service and its relation to the other services. User inputs to the sixth service is shown in red, UI elements are shown in black. The information flow between the sixth service and the others is shown in blue. Accordingly, the user sets up a camera to record a design thinking workshop, then accesses the developed component of the Design2Model tool via the sixth service. A user session is created, and all dependent services are started. The user then configures the Multimedia I service to accept the video stream from the camera setup and performs the workshop. During or after the workshop, key frames are selected for further processing. Images of these selected frames are pre-processed according to requirements of the other services. Based on trained segmentation models, the Image Segmentation service creates masks for areas of interest in the pre-processed images. The user can create custom segmentation models depending on requirements, e.g., to use differently shaped Post-it Notes like arrows, the user must upload labelled images containing those arrow-shaped Post-it Notes to the service for segmentation training. The output of the Image Segmentation service is a mask that is applied by the Multimedia II service to create images containing the areas of interest extracted from the frames selected in the Multimedia I service. E.g., this service creates images for each Post-it Note in a selected frame. Furthermore, a text search is performed, and images of text areas are extracted. Normalization and other means of pre-processing are performed on the results. The images of texts are fed into the Text Recognition service. Afterwards, all the intermediary results are transferred to a service realizing the Digital Design View. This tool shows the workshop results in form of a diagrammatic conceptual model and links all the digital artifacts created by the other services. The result is a digital twin of the design thinking artifacts that is semantically lifted by a conceptual modelling-based transformation. As a result, further model management and processing capabilities are possible. Together, these six services realize the new component of the Design2Model tool.
III. TECHNICAL SOLUTION Different technical solutions are possible for each of the six discussed services. However, they also rely on a common technical structure. Web-based service endpoints make up the interfaces to interact with the services. Each service provides a Swagger specification of a REST API that describes its interfaces. Predefined IP addresses and port ranges are agreed upon and hardware to run the services is selected. Furthermore, a common data storage is adopted by all the services. This allows smooth deployment and operation of the services which employ the following technologies: The Multimedia I service is based on Python3 with the packages Flask, opencv, scipy, imageio and numpy. A previous java-based implementation branch has been abandoned due to efficiency considerations of the engineers working on the service. Currently, the service connects to a locally connected USB camera, records videos and snapshots, and applies image preprocessing.
For the Image Segmentation service, three technical solutions are implemented. The first creates masks based on recurrent neural network technologies, the second employs k-means, and the third makes uses of background subtraction technologies. For the first technical solution, the main environment is Python 3.6, Anaconda, jupyter, MaskRCNN matterport, Tensorflow 1. The Multimedia II service is implemented in Java 1.8 as a Maven 3.5 project making use of the Marvin Image Processing Framework, which provides different algorithms for image manipulation. This allows for the extraction and normalization of regions of interest from an image by combining it with its mask. Furthermore, this service detects and preprocesses areas of handwritten text in the extracted results.
The Text Recognition service is based on Python 3.7.3 with additional packages and Tesseract 4.0. Tesseract supports pipeline-based but also LSTM based algorithms for handwritten text recognition. Additionally, the service provides the option to employ the Microsoft Azure handwriting OCR solution.
Implementing services for the Digital Design View relies on the ADOxx toolsets and XSLT transformations turning native ADOxx model import/export formats into RDF storable in a triple store like Fuseki which supports further processing capabilities. 1 https://austria.omilab.org/psm/content/d2mtool/

IV. EXPERIMENTAL RESULTS
To validate the Design2Model approach and the new Design2Model tool component reported on in this paper, a qualitative analysis is performed. The method employed is artificial ex-post evaluation [12]. The OMiLAB Node in Vienna provides experimentation spaces and settings for design thinking, conceptual modelling, and intelligent operation environments [7]. Building on the resources of the OMiLAB Node Vienna, a set of experiments was conducted in which the new Design2Model tool component is tested during design thinking workshops. This section reports on the results of one such workshop.
The workshop conducted was a Post-it Collaboration Workshop in which a SWOT analysis has been performed for a business idea resulting in new business insights. A poster with clearly distinctive colorful regions for each SWOT category was prepared. After preparing the poster, before starting the workshop, the camera is stabilized on the camera stand and a picture of the empty poster is taken. This image is needed for creating an initial mask used for assigning the single Post-it Notes to the correct SWOT categories. Afterwards, the workshop is conducted, and key frames are selected in the Design2Model component. Fig. 5 shows the poster with the results of the workshop, the empty poster with distinctive regions for each SWOT category, and a mask that is used to assign the individual Post-it Notes to their category in the SWOT analysis. In a next step, the full image is separated into four images, one for each SWOT category. Image segmentation is again applied to identify the individual Post-it Notes on each of the four images. This results in the masks shown in Fig. 6. The resulting masks are processed by the Design2Model tool component in combination with the input image, resulting in images for each Post-it Note for each SWOT category, as shown in Fig. 7. The resulting images are analyzed for text segments and text segments on Post-it Notes are extracted.  The Post-it Notes and the results from text recognition are imported in the Digital Design View of the implemented Design2Model component. This is possible based on a conceptual modelling language defined on the meta-level. The digital workshop results are mapped to this conceptual modelling language. As the modelling language supports diagrammatic representation, a human readable view is produced as shown in Fig. 8. Additionally, the conceptual modelling language provides a formal structure that enables further processing. A simple case of these processing capabilities is also shown in Fig. 8 in which a notebook is opened for one of the modelling elements. The notebook contains the relevant results from text recognition for a Post-it Note, next to other information. Other processing capabilities include model repository management, querying, and import/export in different formats. Fig. 9 shows an excerpt of the RDF view on the conceptual model. This RDF view enables further semantic processing capabilities when stored in a triple store or integrated with Semantic Web and Linked Data frameworks like Apache Jena. Likewise, semantic reasoning capabilities are enables, e.g., actions could be found in an open knowledge based to mitigate weaknesses identified during a SWOT analysis.

V. DISCUSSION
The results of the study revolve around the conceptualization, implementation, and deployment of a new component for the Design2Model tool, which is required to fulfill the objective of OMiLAB. The interpretation of the achieved results is that (1) conceptual modelling can be usefully extended into the powerful methodologies already employed in design environments, which is described by the Design2Model approach, (2) the associated knowledge engineering effort can be mitigated by tooling support for the Design2Model approach, and (3) machine learning offers a set of powerful algorithms to improve tooling and to further reduce the knowledge engineering effort. The new Design2Model tool component implies an advantage for the capabilities of business strategists, decision makers, and designers that have to account for the ongoing disruption in the digital transformation age. However, it is not yet clear how to reliably generalize from the study, as the results have been achieved in a laboratory setting. Additional workshops should take place to collect feedback for further engineering iterations of the new Design2Model tool component.
Preliminary lessons learned include the sensitivity of the Design2Model tool component to illumination conditions during workshops. Either reliable illumination conditions have to be reproduced for each workshop, or a technical solution has to be found. Another factor that heavily influences the success of creating the digital twin of design thinking artifacts is the camera resolution. However, the higher the resolution is, the higher is also the computational cost. More experience is required to find an optimal configuration. Experience also as to be gained on how well end-users can customize the Design2Model tool component by providing training data to the services employing machine learning. From an architectural point of view, the Design2Model tool component relies on a common storage structure which implies a drawback to modularity that can be eliminated in later versions. Another possibility is the collaborative aggregation of different technical implementations of a service to achieve more reliable results. Overall, the results also show that text recognition still has its limits when it comes to bad handwriting.

VI. CONCLUSION
This paper sets out to report on progress made on the Design2Model approach by extending the tooling support available for the approach. A component of the Design2Model tool is conceptualized, implemented, and deployed. The evaluation of the component is performed based on a qualitative analysis of the deployed tool in a laboratory experiment. The validation case is to employ the new Design2Model tool component in a Post-it Collaboration Workshop to support a SWOT analysis from which new business insights can be gained. The result from employing the new Design2Model tool component is a conceptual model of the SWOT analysis supporting further semantic processing and decomposition, providing the potential for realizing the full OMiLAB vision of connecting design environments and operation environments.
Enabling the Design2Model tool component is the combination of machine learning and knowledge engineering. Machine learning is required to provide capabilities to recognize the resulting artifacts of design thinking workshops. In particular, machine learning is employed in the Design2Model tool component to semantically segment areas of interest from an image of a design thinking workshop and for recognizing handwritten text in that image. Knowledge engineering is employed to map the recognition results to semi-formal conceptual structures enabling a knowledge base of diagrammatic conceptual models for human interpretation and automated processing.
Preliminary results are reported at an early stage of the project. Nevertheless, the successful validation of the Design2Model tool component already provides a proof of concept. Future research aims to increase the technical maturity level and to extend the Design2Model tool with additional components.

CONFLICT OF INTEREST
The authors declare no conflict of interest, personal, professional, financial or otherwise.

AUTHOR CONTRIBUTIONS
Michael Walch conducted and supervised the research and wrote the paper under the close guidance of Dimitris Karagiannis considering his continued feedback; all authors had approved the final version.