PDC2024 is the 2024 edition of the Process Discovery Contest.
Do you believe your discovery algorithm can beat the best end score so far of 82.3%?
Then submit it to the PDC 2024 and put it to the test!
The PDC 2024 test contains 288 event logs for which a model needs to be discovered (training logs), and 96 pairs of event logs that are used to evaluate the discovered models (test logs and base logs).
The 96 pairs of test logs and base logs are all generated using the same configurable model, which has the following configurable options:
- Long-term dependencies: yes/no
- Loops: no/simple/complex
- A simple loop has a single point of entry and a single point of exit.
- A complex loop has multiple points of entry and/or multiple points of exit.
- OR constructs: yes/no
- Routing constructs: yes/no
- Optional tasks: yes/no
- Duplicate tasks: yes/no
Each pair is matched by three training logs:
- A training log without noise.
- A training log where in every trace with probability 20% either one random event is removed (40%), moved (20%), or copied (40%).
- Training log 1 with 20 fitting traces classified as positive (boolean
pdc:isPosattribute set totrue) and 20 non-fitting traces classified as negative (booleanpdc:isPosattribute set tofalse).
Each training log contains 1000 traces that result from random walks through the configured model. A discovery algorithm may assume that a trace in a training log is fitting if the boolean pdc:isPos attribute is set to true, and it may assume that it is non-fitting if the boolean pdc:isPos attribute is set to false.
Each test log and each base log contains 1000 traces. For every pair of a test log and a base log we determine for every trace from the test log whether it fits the discovered model better than the corresponding trace from the base log. 500 traces from the test log fit the original model better than the corresponding trace from the base log, and 500 do not. For sake of completeness: Two traces from both logs correspond if and only if they are at the same position in the corresponding logs: The fifth trace of the test log is classified against the fifth trace of the base log, etc.
How, what and when to submit?
Please let Eric Verbeek know that you want to submit your implemented discovery algorithm. Eric will then provide you with a link where you can upload your submission.
You should submit a working discovery algorithm, which can be called using a Discover.bat Windows batch file which takes two parameters:
- The full path to the training log file, excluding the
.xesextension. - The full path to the model file where the discovered model should be stored, excluding any extension like
.pnmlor.bpmn.
As an example, assuming that the miner discovers a Petri net,
Discover.bat logs\discovery\discovery-log models\discovered-model
will discover a model from the training log file logs\discovery\discovery-log.xes and will export the discovered Petri net to the file models\discovered-model.pnml.
If the results of calling your Discovery.bat file as described above is a PNML file (Petri nets) or a BPMN file (BPMN diagram), then you’re done. If not, the discovery algorithm needs to come with its own working classifier algorithm, that is, a Classify.bat Windows batch file, which takes four parameters:
- The full path to the test log file, excluding the
.xesextension. - The full path to the base log file, excluding the
.xesextension. - The full path to the model file which should be used to classify the test log, excluding any extension like
.pnmlor.bpmn. - The full path to the log file where the classified test log should be stored, excluding the
.xesextension.
As an example, assuming that the miner discovered a Petri net,
Classify.bat logs\test\test-log logs\base\base-log models\discovered-model logs\classified\test-log
will classify whether every trace from the test log logs\test\test-log.xes fits the Petri net from models\discovered-model.pnml better than the corresponding trace from the base log logs\base\base-log.xes and it will export the classified traces (by setting the pdc:isPos attribute for every trace) as an event log in logs\classified\test-log.xes.
Classification of a trace is done by adding the boolean pdc:isPos attribute to the trace, which should be:
trueif the trace is classified positive (fits your model better than the corresponding trace in the base model) andfalseif the trace is classified negative (does not fit your model better than the corresponding trace in the base model).
Score and winner
For each training log, the Discovery.bat file is used to discover a model from the training log. Next, the Classify.bat is used to classify every trace in the test log against the corresponding trace in the base log using the discovered model. This results in a positive accuracy rate P and a negative accuracy rate N for this training log. From these, its F-score F is computed as 2*(P*N)/(P+N). The end score for the discovery algorithm is the average F-score over all 288 training logs. However, if computing the F-score for one training log takes on average more than 10 minutes (that is, computing the end score takes more than 2 days), the submission is considered a fail and will have no end score.
The winner is the submission with the best end score.
Key Dates
| Submission deadline | Friday, 20 September 2024 |
| Disclosure of the data set | Saturday, 21 September 2024 |
| Winner notification | Friday, 27 September 2024 |
| Winner announcement | During ICPM 2024 |
Example discovery algorithms
We have run the five example miners on the data set: The Flower miner, the Directly Follows miner, the Trace (or Sequence) miner, the winning miner of the PDC 2023 (PDC2023_Winner), and a non-competing miner that scored best on the PDC 2023 data set (PDC2023_Best). The following table shows the results for these miners.
| Miner | Positive accuracy | Negative accuracy | End score | Average time for F-score (in seconds) |
|---|---|---|---|---|
| Directly Follows | 25.6% | 91.2% | 37.1% | 47 |
| Flower | 0.0% | 100.0% | 0.0% | 29 |
| Trace | 66.6% | 88.0% | 75.7% | 36 |
| PDC2023_Winner | 58.0% | 95.9% | 69.9% | 16 |
| PDC2023_Best | 75.7% | 94.3% | 82.3% | 268 |
Process Discovery Contest Workshop
This year, there will be a workshop around the Process Discovery Contest: The Process Discovery Contest Workshop (PDCW) 2024. In this workshop, every contestant gets to introduce the discovery approach they are using, and to show and discuss discovery results on a number of selected event logs from the contest (preferably, using a live demo). This provides an overview of the current state in the field of (automated) process discovery and where there is room for improvement. It also shows stronger and weaker points of the different approaches. Prior to these presentations, an overview of how the current Process Discovery Contest has been set up will be presented, as well as the ground truth models for the selected event logs.
More details will follow later.
PDCW 2024 Audience award
The attendees of the PDCW 2024 workshop get to vote on what they think is the best submission. The submission that gets the most votes, wins the PDCW 2024 audience award.
Voting rules
- Everybody that attends a presentation on a submission, will obtain one token labeled with that submission.
- At the end of the workshop, everybody can cast as many votes as s/he has tokens.
- A token labeled with a certain submission cannot be used to vote for that submission, it can only be used to vote on a different submission.