Conformance Checking Challenge 2019

The 1st Conformance Checking Challenge 2019 (CCC19) is a co-located event of the 1st International Conference on Process Mining 2019. The Challenge provides participants with artifacts stemmed from a real process, and invites them to analyze the conformance between the observed (event log) and expected behaviour (model) of the process, with an emphasis on providing process owners with interpretable and understandable conformance results, in terms of a report. We further encourage participants to use any tools or technique at their disposal, including open-sourced or proprietary ones, as well as those implemented specifically for this challenge.

THE DATES

Abstract submission deadline:May 4, 2019,

with submission
Report submission deadline:May 11, 2019

May 18, 2019
Presentation of the winners:At the ICPM Conference dinner, Aachen, Germany
Conference dates:24 - 26 June 2019, Aachen, Germany

THE CATEGORIES

The Challenge is divided in 3 categories: Student, Academic, and Professional.

Student

This category targets Bachelor, Master and PhD students or student teams. In this category, the focus is on the originality of the results, the validity of the claims, the interpretability of the conclusions, and the depth of the conformance analysis. Participants must include representative analysis for the 2 perspectives proposed (see Evaluation Section), but developing or implementing specifics tools is not required and they could use well-known existing ones.

We received the following submissions in the Student category:

Academic

This category targets academics and the focus is much more on the novelty of the techniques applied and the understandability of the results, than the actual results. This provides a great opportunity for BPI researchers to show the practical applicability of their tools and/or techniques on a real process. Analysis can include 1 or both perspective (see Evaluation Section).

We received the following submissions in the Academic category:

Professional

This category targets professionals to show their skills in analyzing conformance and tool vendors to demonstrate the functionalities of their tools. The submitted reports will be judged on their level of professionalism, and must include the 2 perspectives proposed (see Evaluation Section). The participants are expected to report on a broader range of aspects, where each aspect does not have to be developed in full detail. The report submitted in this category will be judged on its completeness of analysis and usefulness for the purpose of a real-life business improvement setting.

We received the following submissions in the Professional category:

THE PROCESS

This edition of the Challenge will focus on a medical training process. In particular, the process captures the procedure used for installing Central Venous Catheters (CVC) with ultrasound. Moreover, this process is the one used by doctors for training medical students in this specific task. The CVC procedure refers to installing a catheter (tube) in a central vein, which is then used for delivering liquids, fluids, or medicines to patients. The general idea of the procedure is the following:

  1. The patient is prepared, the target vein is identified with ultrasound, and the patient is anesthetized.
  2. A large hollow needle (Trocar), attached to a syringe, is inserted into the vein with the help of ultrasound.
  3. The Syringe is used for checking for a blood return, implying that the trocar has been correctly installed. If so, the syringe can be safely removed from the trocar.
  4. A Guidewire is inserted at about 30 cm through the trocar. Then, the trocar is removed.
  5. The pathway is widen and the wire is used for guiding the insertion of the Catheter.
  6.  Finally, the guidewire is removed and the the final position of the catheter is checked.

The data was collected during a course delivery from an institution that teaches future doctors how to perform a CVC installation. The training is structured as follows:

  • Students are taught how to perform the procedure by their instructors
  • Students take a first preliminary test on the procedure (PRE)
  • Students can practice the procedure at their discretion
  • Students take a final post assessment (POST) to show they have acquired the skills required by the procedure.

The dataset provided with this challenge include the event log corresponding to execution of PRE and POST tests performed by 10 students.

The dataset has been collected as part of the multidisciplinary research project “Process-Oriented Medical Education (POME)” conducted by the School of Medicine and the School of Engineering of the Pontificia Universidad Católica de Chile. For citation and more information we refer to the document included in the dataset.

THE DATASET

The dataset of the Challenge is available as public dataset with doi doi:10.4121/uuid:c923af09-ce93-44c3-ace0-c5508cf103ad in the 4TU Data Repository, and can be downloaded here (2MB) and from http://doi.org/10.4121/uuid:c923af09-ce93-44c3-ace0-c5508cf103ad.

When using this data, please refer to it as:

Munoz-Gama, J., de la Fuente, R., Sepúlveda, M., Fuentes, R., 
Conformance Checking Challenge 2019. 4TU.Centre for Research Data. 
https://doi.org/10.4121/uuid:c923af09-ce93-44c3-ace0-c5508cf103ad

The dataset includes:

  • Description: A short slidedeck with the description of the Challenge and pointers to more information.
  • Model: A Delphi method was designed to establish clinical consensus for medical procedures (and processes in general), and it was used to define a BPMN model for the Central Venous Catheter installation with ultrasound. The Delphi panel was answered by 13 experts from 3 medical specialties and 8 medical institutions. The dataset contains the model in BPMN notation, Petri Net notation, in PNG, and Natural Text. Models in other formats derived from those are also welcomed in the Challenge.
  • Log: A video was recorded of each execution (PRE and POST for each 10 students). A special software was designed to tag the videos with the activity performed, and their initial and final times. Since it is a manual tagging, the data may contain noise. The dataset contains the log in CVS notation, XES notation, and XLSX The log contains 20 cases, 697 events, 29 activities. The log contains the following information per event:
    • CASEID: Trace ID
    • RESOURCE: Student id executing the procedure
    • ROUND: ‘Pre’ if it was performed by prior the practice sessions (PRE), or ‘Post’ if it was after the practice sessions (POST).
    • EVENTID: Event id
    • ACTIVITY: Name of the activity of the process performed.
    • STAGE: Stage of the procedure the activity belongs to (Operator and Patient Preparation, Ultrasound Preparation, Locate Structures, Venous puncture, Install Guidewire, or Install Catheter).
    • START: Time when the activity started
    • END: Time when the activity was finished
    • VIDEOSTART: Time in the video recording when the activity started
    • VIDEOEND: Time in the video recording when the activity finished

THE REPORT

A submission should contain a PDF Report of at most 25 pages, including figures, using the LNCS/LNBIP format (http://www.springer.com/computer/lncs?SGWID=0-164-6-791344-0) specified by Springer (available for both LaTeX and MS Word). Appendices may be included, but should only support the main text. Submissions must be made through EasyChair at https://easychair.org/conferences/?conf=icpm2019 where you must indicate that your submission is to the Conformance Checking Challenge.

THE EVALUATION

Being the process in a training course context, we consider two perspectives to include in the conformance report, one for each stakeholder: Students and Instructor.

  • Student’s Perspective: Primarily interested on their own performance, a student would aim at identifying his/her own mistakes, looking at improving his/her overall execution before the POST test.
  • Instructor’s Perspective: Aiming at an aggregated view of the performance of the group, an instructor would try to identify common mistakes, similarities and differences exhibit by the group of students, the need for additional training, so on and so forth.

Reports in the Student Category and Professional Category (see The Categories Section) should include both perspectives, while Academic Category reports must clearly state which perspective are they analyzing (or both). Concerning the student perspective, it is not necessary to include the analysis of the 10 individual students, but illustrative examples of feedback, as well as hints on how such feedback could benefit the student under analysis.

Being the process in a interdisciplinary context, there is a strong emphasis on interpretability of the conformance report. Particularly, we aim at feedback that could be interpreted by end users in the domain, i.e. doctors and medical students. With this in mind, the submissions will be evaluated according to the following criteria:

  • Understandability: The conformance results is easy to understand and interpret for end users in the domain.
  • Insightfulness: The conformance feedback is insightful for the defined stakeholders.
  • Originality: The report proposes novel techniques, or uses existing techniques in an original and interesting way for these specific scenario.

The reports will be evaluated by a group of experts on conformance checking and surgical domains.

ORGANIZERS

Abel Armas-Cervantes – The University of Melbourne
Luciano García-Bañuelos – Tecnológico de Monterrey
Jorge Munoz-Gama – Pontificia Universidad Católica de Chile

Any questions related to the contest can be directed to the organizers via conformancechallenge@gmail.com