diff --git a/spacy_ner_convert-iob-format/_learning_brick.yaml b/spacy_ner_convert-iob-format/_learning_brick.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..a185cf4c8de0e3a6f235aa6a45f6085f96c9af65
--- /dev/null
+++ b/spacy_ner_convert-iob-format/_learning_brick.yaml
@@ -0,0 +1,51 @@
+---
+brick-version: 0.1
+
+title: "spaCy NER: Konvertieren des Outputs in IOB-Format für Evaluation"
+shorttitle: "spaCy Konvertieren in IOB-Format"
+
+description: >-
+ Das Notebook zeigt, wie der Output der NER durch spaCy in das IOB-Format
+ umgewandelt werden kann, um eine Evaluation auf der Ground Truth durchzuführen.
+tags:
+ - "ner"
+ - "nlp"
+ - "spacy"
+ - "iob"
+
+authors:
+ - name: "Martin Dröge"
+ orcid: optional
+ email: optional
+ url: optional
+ role: author
+ affiliation:
+ - name: "Humboldt Universität zu Berlin"
+ institution: "Institut für Geschichstwissenschaft"
+ department: "Philosophische Fakultät"
+ url: "https://www.hu-berlin.de"
+
+computational-notebook:
+ platform: "jupyter"
+ language: "python"
+ audience-level: "beginner"
+ date: "2024-01-31"
+ requirements:
+ pip: "requirements.txt"
+ apt: "apt.txt"
+
+repository:
+ url: "https://scm.cms.hu-berlin.de/digital-history/lehre/learning-bricks/"
+ issues: "https://scm.cms.hu-berlin.de/digital-history/lehre/learning-bricks/issues/"
+
+publication:
+ url: "https://www.example.com/spacy-basics/"
+ doi: "10.5281/zenodo.XYZ_EXAMPLE"
+ license: CC-BY-4.0
+
+data:
+ path:
+ - "/example-news-text.txt"
+ - "/ground-truth-example-news-text.conll"
+ doi: "10.5281/zenodo.XYZ_EXAMPLE"
+ license: CC-BY-4.0
diff --git a/spacy_ner_convert-iob-format/example-news-text.txt b/spacy_ner_convert-iob-format/example-news-text.txt
new file mode 100644
index 0000000000000000000000000000000000000000..8d2b08411a01dca37a6bc9f3482524d944fcbf16
--- /dev/null
+++ b/spacy_ner_convert-iob-format/example-news-text.txt
@@ -0,0 +1,68 @@
+Konflikt um Kaukasus-Region : Wie die Diplomatie bei Bergkarabach versagt
+von Nina Niebergall, Eriwan
+
+Datum:
+05.10.2023 14:29 Uhr
+
+Die Welt konnte oder wollte nicht helfen, als hunderttausend Armenier aus Bergkarabach vertrieben wurden. Der Konflikt ist nicht vorbei. Doch die Diplomatie hat keine Lösungen.
+Ein Blick auf Khankendi in Aserbaidschan, das den Armeniern auch als Stepanakert, Karabach, bekannt ist.
+Der Konflikt um Bergkarabach ist längst nicht vorbei.
+Quelle: dpa
+
+Die Armenier*innen aus Bergkarabach, die man in diesen Tagen in Eriwan, der Hauptstadt Armeniens antrifft, sind erschöpft, traurig. Aber vor allem wütend. Wahlweise auf die armenische Regierung, auf Aserbaidschan, auf Russland, die UN oder die EU. Denn die Enttäuschungen häufen sich, von allen Seiten.
+
+Eigentlich sollten sich an diesem Donnerstag der armenische Premier Nikol Paschinjan und der aserbaidschanische Präsident Ilham Alijew im spanischen Granada treffen. Es wäre nach dem Angriff Aserbaidschans auf Bergkarabach, nach dem Massenexodus von mehr als hunderttausend Menschen aus Bergkarabach zumindest eine Rückkehr an den Verhandlungstisch gewesen. Womöglich sogar die Chance auf einen fragilen Frieden.
+Aserbaidschan sagt Treffen in Granada ab
+
+Doch Alijew sagte das Treffen am Vortag ab. Laut aserbaidschanischen Medien war der autokratische Präsident nicht glücklich mit der geplanten Zusammensetzung des Treffens, an dem auch Bundeskanzler Olaf Scholz, Frankreichs Präsident Emmanuel Maron und EU-Ratspräsident Charles Michel hätten teilnehmen sollen.
+
+Mehr als 100.000 Armenier und Armenierinnen sind aus Bergkarabach geflohen.
+
+Alijew störten offenbar zwei Punkte: Erstens, dass die Türkei nicht mit am Tisch sitzt, der engste Verbündete. Und zweitens, dass Frankreich dabei sein sollte. Am Mittwoch war die französische Außenministerin in Eriwan und erklärte, ihr Land sei offen für Waffenlieferungen, damit Armenien seine Verteidigung sicherstellen könne.
+Worum es bei Verhandlungen gehen könnte
+
+Marcel Röthig, Leiter des Regionalbüros Südkaukasus von der Friedrich-Ebert-Stiftung, vermutet, dass das Treffen in ähnlicher Zusammensetzung in den kommenden Monaten nachgeholt werde. Aber: Um Bergkarabach gehe es bei Verhandlungen kaum noch. "Vielmehr dürfte es jetzt um die gegenseitige Anerkennung der Grenze gehen, um ein Ausweiten des Konfliktes auf armenisches Territorium zu vermeiden", so Röthig.
+
+Aber ich bin realistisch. Ich kann mir nicht vorstellen, dass man jetzt noch von einem gerechten Frieden sprechen kann. Nicht, nachdem Baku den Weg der Diplomatie einmal verlassen hat.
+
+Marcel Röthig, Südkaukasus-Experte
+
+Aserbaidschan hat in den vergangenen Jahren immer wieder auch armenisches Territorium angegriffen. Alijew, ebenso wie sein türkischer Partner Recep Tayyip Erdogan äußern offen Pläne über Bergkarabach hinaus. Dabei geht es um eine Landverbindung zwischen Aserbaidschan und der Türkei, die mitten durch den Süden Armeniens führen würde. Und die Aserbaidschan sich mit Waffengewalt erkämpfen müsste.
+
+Es wäre eine Eskalation, die einen anderen geopolitischen Player auf den Plan rufen könnte: Den Iran. Denn eine Landverbindung zwischen der Türkei und Aserbaidschan würde den Iran von Armenien abschneiden. Da hat Teheran eine rote Linie gezogen.
+
+Armenien möchte dem Internationalen Strafgerichtshof beitreten, eine Abkehr von Russland. Zur Reaktion vom Kreml eine Einschätzung von ZDF-Korrespondent Armin Coerper.
+
+Russland hat kaum noch Einfluss
+
+Die Diplomatie wird also durchaus weiter gebraucht. Aber wer hat noch Einfluss in der Region, wer vertritt armenische Interessen? Russland galt traditionell als Schutzmacht Armeniens.
+
+Wladimir Putin hatte in der Vergangenheit immer wieder Gesprächsformate zwischen Paschinjan und Alijew organisiert. Doch aktuell ist es unwahrscheinlich, dass Russland eine größere Rolle spielen wird. Armenien hat sich abgewendet von Moskau, das längst nicht mehr als verlässlicher Partner wahrgenommen wird.
+
+Armenien hat sich zuletzt von Russland abgewendet – auch durch den Beitritt zum Internationalen Gerichtshof:
+
+Hunderttausend Armenier*innen sind aus Bergkarabach geflohen, fühlen sich von Moskau im Stich gelassen. Nun tritt Armenien dem Internationalen Gerichtshof bei – eine Abkehr von Russland.
+
+Russland schürt mit Propaganda gezielt die armenische Wut auf ihren Premier, der Bergkarabach aus Sicht einiger Armenier*innen aufgegeben hat.
+
+ Die Russen lassen Aserbaidschan gewähren, in der Hoffnung, dass es Paschinjans Machtbasis destabilisiert. Fällt er, würde eine Nachfolge möglicherweise Armenien zurück in den Moskauer Orbit führen.
+
+Deal zwischen Putin und Erdogan?
+
+Expert*innen gehen davon aus, dass es einen Deal zwischen Putin und Erdogan gab, den großen Playern in der Region, bevor Aserbaidschan Bergkarabach angriff. Auch Marcel Röthig hält das für wahrscheinlich.
+
+ In den vergangenen Jahren haben Russland und die Türkei oft genug Interessen abgesteckt. Offensichtlich war er bei Syrien so, jetzt in Armenien und ich glaube zum Teil auch in der Ukraine.
+
+Es ist nicht auszuschließen, dass Russland, die Türkei, der Iran und Aserbaidschan eine gemeinsame Lösung finden, die die Interessen aller berücksichtigt. Außer von Armenien. Das kleine Land steht sehr allein da.
+
+Fast zwei Wochen nach der Eroberung durch Aserbaidschan hat ein Großteil der 120.000 armenischen Einwohner Bergkarabach verlassen. Sebastian Ehm über die Situation vor Ort.
+
+EU zwischen Gasdeal und Moral
+
+Und die Europäer? EU-Kommissionspräsidentin Ursula von der Leyen bekam viel Spott auf der Plattform "X", als sie den Armenier*innen Unterstützung verspricht, aber den Aggressor nicht nennt. Denn die EU hat mit Aserbaidschan einen Gasdeal geschlossen, nachdem Russland als Lieferant ausfiel.
+
+Südkaukasus-Experte Röthig hält es trotzdem für wahrscheinlich, dass Deutschland sich stärker an die Seite Armeniens stellt, etwa mit Visa-Liberalisierungen, einer ausgeweiteten EU-Beobachtermission und einer klaren Sprache gegenüber Aserbaidschan.
+
+Doch es ist völlig unklar, ob das reicht, um Aserbaidschan etwas entgegenzusetzen, das zerbrochene Vertrauen der Armenier*innen zurückzugewinnen.
+
+Nina Niebergall berichtet als ZDF-Korrespondentin über Russland, die Kaukasusregion und Zentralasien.
diff --git a/spacy_ner_convert-iob-format/ground-truth-example-news-text.conll b/spacy_ner_convert-iob-format/ground-truth-example-news-text.conll
new file mode 100644
index 0000000000000000000000000000000000000000..e0b565a752650e2f8d0594e65961d45ac9980554
--- /dev/null
+++ b/spacy_ner_convert-iob-format/ground-truth-example-news-text.conll
@@ -0,0 +1,1149 @@
+Konflikt O
+um O
+Kaukasus-Region B-LOC
+: O
+Wie O
+die O
+Diplomatie O
+bei O
+Bergkarabach B-LOC
+versagt O
+
+ O
+von O
+Nina B-PER
+Niebergall I-PER
+, O
+Eriwan B-LOC
+
+
+ O
+Datum O
+: O
+
+ O
+05.10.2023 O
+14:29 O
+Uhr O
+
+
+ O
+
+Die O
+Welt O
+konnte O
+oder O
+wollte O
+nicht O
+helfen O
+, O
+als O
+hunderttausend O
+Armenier O
+aus O
+Bergkarabach B-LOC
+vertrieben O
+wurden O
+. O
+
+Der O
+Konflikt O
+ist O
+nicht O
+vorbei O
+. O
+
+Doch O
+die O
+Diplomatie O
+hat O
+keine O
+Lösungen O
+. O
+
+ O
+
+Ein O
+Blick O
+auf O
+Khankendi B-LOC
+in O
+Aserbaidschan B-LOC
+, O
+das O
+den O
+Armeniern O
+auch O
+als O
+Stepanakert B-LOC
+, O
+Karabach B-LOC
+, O
+bekannt O
+ist O
+. O
+
+ O
+
+Der O
+Konflikt O
+um O
+Bergkarabach B-LOC
+ist O
+längst O
+nicht O
+vorbei O
+. O
+
+ O
+
+Quelle O
+: O
+dpa B-ORG
+
+
+ O
+
+Die O
+Armenier*innen O
+aus O
+Bergkarabach B-LOC
+, O
+die O
+man O
+in O
+diesen O
+Tagen O
+in O
+Eriwan B-LOC
+, O
+der O
+Hauptstadt O
+Armeniens B-LOC
+antrifft O
+, O
+sind O
+erschöpft O
+, O
+traurig O
+. O
+
+Aber O
+vor O
+allem O
+wütend O
+. O
+
+Wahlweise O
+auf O
+die O
+armenische O
+Regierung O
+, O
+auf O
+Aserbaidschan B-LOC
+, O
+auf O
+Russland B-LOC
+, O
+die O
+UN B-ORG
+oder O
+die O
+EU B-ORG
+. O
+
+Denn O
+die O
+Enttäuschungen O
+häufen O
+sich O
+, O
+von O
+allen O
+Seiten O
+. O
+
+
+ O
+
+Eigentlich O
+sollten O
+sich O
+an O
+diesem O
+Donnerstag O
+der O
+armenische O
+Premier O
+Nikol B-PER
+Paschinjan I-PER
+und O
+der O
+aserbaidschanische O
+Präsident O
+Ilham B-PER
+Alijew I-PER
+im O
+spanischen O
+Granada B-LOC
+treffen O
+. O
+
+Es O
+wäre O
+nach O
+dem O
+Angriff O
+Aserbaidschans B-LOC
+auf O
+Bergkarabach B-LOC
+, O
+nach O
+dem O
+Massenexodus O
+von O
+mehr O
+als O
+hunderttausend O
+Menschen O
+aus O
+Bergkarabach B-LOC
+zumindest O
+eine O
+Rückkehr O
+an O
+den O
+Verhandlungstisch O
+gewesen O
+. O
+
+Womöglich O
+sogar O
+die O
+Chance O
+auf O
+einen O
+fragilen O
+Frieden O
+. O
+
+ O
+
+Aserbaidschan B-LOC
+sagt O
+Treffen O
+in O
+Granada B-LOC
+ab O
+
+
+ O
+
+Doch O
+Alijew B-PER
+sagte O
+das O
+Treffen O
+am O
+Vortag O
+ab O
+. O
+
+Laut O
+aserbaidschanischen O
+Medien O
+war O
+der O
+autokratische O
+Präsident O
+nicht O
+glücklich O
+mit O
+der O
+geplanten O
+Zusammensetzung O
+des O
+Treffens O
+, O
+an O
+dem O
+auch O
+Bundeskanzler O
+Olaf B-PER
+Scholz I-PER
+, O
+Frankreichs B-LOC
+Präsident O
+Emmanuel B-PER
+Maron I-PER
+und O
+EU-Ratspräsident O
+Charles B-PER
+Michel I-PER
+hätten O
+teilnehmen O
+sollen O
+. O
+
+
+ O
+
+Mehr O
+als O
+100.000 O
+Armenier O
+und O
+Armenierinnen O
+sind O
+aus O
+Bergkarabach B-LOC
+geflohen O
+. O
+
+
+ O
+
+Alijew B-PER
+störten O
+offenbar O
+zwei O
+Punkte O
+: O
+
+Erstens O
+, O
+dass O
+die O
+Türkei B-LOC
+nicht O
+mit O
+am O
+Tisch O
+sitzt O
+, O
+der O
+engste O
+Verbündete O
+. O
+
+Und O
+zweitens O
+, O
+dass O
+Frankreich B-LOC
+dabei O
+sein O
+sollte O
+. O
+
+Am O
+Mittwoch O
+war O
+die O
+französische O
+Außenministerin O
+in O
+Eriwan B-LOC
+und O
+erklärte O
+, O
+ihr O
+Land O
+sei O
+offen O
+für O
+Waffenlieferungen O
+, O
+damit O
+Armenien B-LOC
+seine O
+Verteidigung O
+sicherstellen O
+könne O
+. O
+
+ O
+
+Worum O
+es O
+bei O
+Verhandlungen O
+gehen O
+könnte O
+
+
+ O
+Marcel B-PER
+Röthig I-PER
+, O
+Leiter O
+des O
+Regionalbüros O
+Südkaukasus B-LOC
+von O
+der O
+Friedrich-Ebert-Stiftung B-ORG
+, O
+vermutet O
+, O
+dass O
+das O
+Treffen O
+in O
+ähnlicher O
+Zusammensetzung O
+in O
+den O
+kommenden O
+Monaten O
+nachgeholt O
+werde O
+. O
+
+Aber O
+: O
+Um O
+Bergkarabach B-LOC
+gehe O
+es O
+bei O
+Verhandlungen O
+kaum O
+noch O
+. O
+
+" O
+Vielmehr O
+dürfte O
+es O
+jetzt O
+um O
+die O
+gegenseitige O
+Anerkennung O
+der O
+Grenze O
+gehen O
+, O
+um O
+ein O
+Ausweiten O
+des O
+Konfliktes O
+auf O
+armenisches O
+Territorium O
+zu O
+vermeiden O
+" O
+, O
+so O
+Röthig B-PER
+. O
+
+
+ O
+
+Aber O
+ich O
+bin O
+realistisch O
+. O
+
+Ich O
+kann O
+mir O
+nicht O
+vorstellen O
+, O
+dass O
+man O
+jetzt O
+noch O
+von O
+einem O
+gerechten O
+Frieden O
+sprechen O
+kann O
+. O
+
+Nicht O
+, O
+nachdem O
+Baku B-LOC
+den O
+Weg O
+der O
+Diplomatie O
+einmal O
+verlassen O
+hat O
+. O
+
+
+ O
+
+Marcel B-PER
+Röthig I-PER
+, O
+Südkaukasus-Experte O
+
+
+ B-LOC
+Aserbaidschan I-LOC
+hat O
+in O
+den O
+vergangenen O
+Jahren O
+immer O
+wieder O
+auch O
+armenisches O
+Territorium O
+angegriffen O
+. O
+
+Alijew B-PER
+, O
+ebenso O
+wie O
+sein O
+türkischer O
+Partner O
+Recep B-PER
+Tayyip I-PER
+Erdogan I-PER
+äußern O
+offen O
+Pläne O
+über O
+Bergkarabach B-LOC
+hinaus O
+. O
+
+Dabei O
+geht O
+es O
+um O
+eine O
+Landverbindung O
+zwischen O
+Aserbaidschan B-LOC
+und O
+der O
+Türkei B-LOC
+, O
+die O
+mitten O
+durch O
+den O
+Süden O
+Armeniens B-LOC
+führen O
+würde O
+. O
+
+Und O
+die O
+Aserbaidschan B-LOC
+sich O
+mit O
+Waffengewalt O
+erkämpfen O
+müsste O
+. O
+
+
+ O
+
+Es O
+wäre O
+eine O
+Eskalation O
+, O
+die O
+einen O
+anderen O
+geopolitischen O
+Player O
+auf O
+den O
+Plan O
+rufen O
+könnte O
+: O
+Den O
+Iran B-LOC
+. O
+
+Denn O
+eine O
+Landverbindung O
+zwischen O
+der O
+Türkei B-LOC
+und O
+Aserbaidschan B-LOC
+würde O
+den O
+Iran B-LOC
+von O
+Armenien B-LOC
+abschneiden O
+. O
+
+Da O
+hat O
+Teheran B-LOC
+eine O
+rote O
+Linie O
+gezogen O
+. O
+
+
+ O
+
+Armenien B-LOC
+möchte O
+dem O
+Internationalen B-ORG
+Strafgerichtshof I-ORG
+beitreten O
+, O
+eine O
+Abkehr O
+von O
+Russland B-LOC
+. O
+
+Zur O
+Reaktion O
+vom O
+Kreml B-LOC
+eine O
+Einschätzung O
+von O
+ZDF-Korrespondent O
+Armin B-PER
+Coerper I-PER
+. O
+
+
+ O
+
+Russland B-LOC
+hat O
+kaum O
+noch O
+Einfluss O
+
+
+ O
+
+Die O
+Diplomatie O
+wird O
+also O
+durchaus O
+weiter O
+gebraucht O
+. O
+
+Aber O
+wer O
+hat O
+noch O
+Einfluss O
+in O
+der O
+Region O
+, O
+wer O
+vertritt O
+armenische O
+Interessen O
+? O
+
+Russland B-LOC
+galt O
+traditionell O
+als O
+Schutzmacht O
+Armeniens B-LOC
+. O
+
+
+ O
+
+Wladimir B-PER
+Putin I-PER
+hatte O
+in O
+der O
+Vergangenheit O
+immer O
+wieder O
+Gesprächsformate O
+zwischen O
+Paschinjan B-PER
+und O
+Alijew B-PER
+organisiert O
+. O
+
+Doch O
+aktuell O
+ist O
+es O
+unwahrscheinlich O
+, O
+dass O
+Russland B-LOC
+eine O
+größere O
+Rolle O
+spielen O
+wird O
+. O
+
+Armenien B-LOC
+hat O
+sich O
+abgewendet O
+von O
+Moskau B-LOC
+, O
+das O
+längst O
+nicht O
+mehr O
+als O
+verlässlicher O
+Partner O
+wahrgenommen O
+wird O
+. O
+
+
+ O
+
+Armenien B-LOC
+hat O
+sich O
+zuletzt O
+von O
+Russland B-LOC
+abgewendet O
+– O
+auch O
+durch O
+den O
+Beitritt O
+zum O
+Internationalen B-ORG
+Gerichtshof I-ORG
+: O
+
+
+ O
+
+Hunderttausend O
+Armenier*innen O
+sind O
+aus O
+Bergkarabach B-LOC
+geflohen O
+, O
+fühlen O
+sich O
+von O
+Moskau B-LOC
+im O
+Stich O
+gelassen O
+. O
+
+Nun O
+tritt O
+Armenien B-LOC
+dem O
+Internationalen B-ORG
+Gerichtshof I-ORG
+bei O
+– O
+eine O
+Abkehr O
+von O
+Russland B-LOC
+. O
+
+
+ O
+
+Russland B-LOC
+schürt O
+mit O
+Propaganda O
+gezielt O
+die O
+armenische O
+Wut O
+auf O
+ihren O
+Premier O
+, O
+der O
+Bergkarabach B-LOC
+aus O
+Sicht O
+einiger O
+Armenier*innen O
+aufgegeben O
+hat O
+. O
+
+
+ O
+
+Die O
+Russen O
+lassen O
+Aserbaidschan B-LOC
+gewähren O
+, O
+in O
+der O
+Hoffnung O
+, O
+dass O
+es O
+Paschinjans B-ORG
+Machtbasis I-ORG
+destabilisiert O
+. O
+
+Fällt O
+er O
+, O
+würde O
+eine O
+Nachfolge O
+möglicherweise O
+Armenien B-LOC
+zurück O
+in O
+den O
+Moskauer O
+Orbit O
+führen O
+. O
+
+
+ O
+
+Deal O
+zwischen O
+Putin B-PER
+und O
+Erdogan B-PER
+? O
+
+
+ O
+
+Expert*innen O
+gehen O
+davon O
+aus O
+, O
+dass O
+es O
+einen O
+Deal O
+zwischen O
+Putin B-PER
+und O
+Erdogan B-PER
+gab O
+, O
+den O
+großen O
+Playern O
+in O
+der O
+Region O
+, O
+bevor O
+Aserbaidschan B-LOC
+Bergkarabach B-LOC
+angriff O
+. O
+
+Auch O
+Marcel B-PER
+Röthig I-PER
+hält O
+das O
+für O
+wahrscheinlich O
+. O
+
+
+ O
+
+In O
+den O
+vergangenen O
+Jahren O
+haben O
+Russland B-LOC
+und O
+die O
+Türkei B-LOC
+oft O
+genug O
+Interessen O
+abgesteckt O
+. O
+
+Offensichtlich O
+war O
+er O
+bei O
+Syrien B-LOC
+so O
+, O
+jetzt O
+in O
+Armenien B-LOC
+und O
+ich O
+glaube O
+zum O
+Teil O
+auch O
+in O
+der O
+Ukraine B-LOC
+. O
+
+
+ O
+
+Es O
+ist O
+nicht O
+auszuschließen O
+, O
+dass O
+Russland B-LOC
+, O
+die O
+Türkei B-LOC
+, O
+der O
+Iran B-LOC
+und O
+Aserbaidschan B-LOC
+eine O
+gemeinsame O
+Lösung O
+finden O
+, O
+die O
+die O
+Interessen O
+aller O
+berücksichtigt O
+. O
+
+Außer O
+von O
+Armenien B-LOC
+. O
+
+Das O
+kleine O
+Land O
+steht O
+sehr O
+allein O
+da O
+. O
+
+
+ O
+
+Fast O
+zwei O
+Wochen O
+nach O
+der O
+Eroberung O
+durch O
+Aserbaidschan B-LOC
+hat O
+ein O
+Großteil O
+der O
+120.000 O
+armenischen O
+Einwohner O
+Bergkarabach B-LOC
+verlassen O
+. O
+
+Sebastian B-PER
+Ehm I-PER
+über O
+die O
+Situation O
+vor O
+Ort O
+. O
+
+
+ O
+
+EU B-ORG
+zwischen O
+Gasdeal O
+und O
+Moral O
+
+
+ O
+
+Und O
+die O
+Europäer O
+? O
+EU-Kommissionspräsidentin O
+Ursula B-PER
+von I-PER
+der I-PER
+Leyen I-PER
+bekam O
+viel O
+Spott O
+auf O
+der O
+Plattform O
+" O
+X B-ORG
+" O
+, O
+als O
+sie O
+den O
+Armenier*innen O
+Unterstützung O
+verspricht O
+, O
+aber O
+den O
+Aggressor O
+nicht O
+nennt O
+. O
+
+Denn O
+die O
+EU B-ORG
+hat O
+mit O
+Aserbaidschan B-LOC
+einen O
+Gasdeal O
+geschlossen O
+, O
+nachdem O
+Russland B-LOC
+als O
+Lieferant O
+ausfiel O
+. O
+
+
+ O
+
+Südkaukasus-Experte O
+Röthig B-PER
+hält O
+es O
+trotzdem O
+für O
+wahrscheinlich O
+, O
+dass O
+Deutschland B-LOC
+sich O
+stärker O
+an O
+die O
+Seite O
+Armeniens B-LOC
+stellt O
+, O
+etwa O
+mit O
+Visa-Liberalisierungen O
+, O
+einer O
+ausgeweiteten O
+EU-Beobachtermission O
+und O
+einer O
+klaren O
+Sprache O
+gegenüber O
+Aserbaidschan B-LOC
+. O
+
+
+ O
+
+Doch O
+es O
+ist O
+völlig O
+unklar O
+, O
+ob O
+das O
+reicht O
+, O
+um O
+Aserbaidschan B-LOC
+etwas O
+entgegenzusetzen O
+, O
+das O
+zerbrochene O
+Vertrauen O
+der O
+Armenier*innen O
+zurückzugewinnen O
+. O
+
+
+ O
+
+Nina B-PER
+Niebergall I-PER
+berichtet O
+als O
+ZDF-Korrespondentin O
+über O
+Russland B-LOC
+, O
+die O
+Kaukasusregion B-LOC
+und O
+Zentralasien B-LOC
+. O
+
diff --git a/spacy_ner_convert-iob-format/spacy-ner-convert-iob-format.ipynb b/spacy_ner_convert-iob-format/spacy-ner-convert-iob-format.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..ea555a92f0f6a723aa3e2a377e237b0a609e1ff7
--- /dev/null
+++ b/spacy_ner_convert-iob-format/spacy-ner-convert-iob-format.ipynb
@@ -0,0 +1,1760 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "503b694c-648c-40ae-ba99-e79fdff66695",
+ "metadata": {},
+ "source": [
+ "# spaCy NER: Konvertieren des Outputs in das IOB-Format zur Evaluation\n",
+ "\n",
+ "Die nachfolgenden Inhalte bauen auf den Grundlagen zu [spaCy](https://spacy.io/) auf, die im Notebook [spaCy basics](https://scm.cms.hu-berlin.de/digital-history/lehre/learning-bricks/-/blob/c110a2481ccbf970ea24bac9d434c65643d4b7be/spacy_basics/spacy-basics.ipynb) zu finden sind. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eb965ed1-8c7f-4271-8994-7a85c020b883",
+ "metadata": {},
+ "source": [
+ "## Import\n",
+ "\n",
+ "Nach dem Import der benötigten Bibliotheken wird die Version gecheckt."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "528e1c65-cfa0-4c1d-90da-61f70aa714e1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import spacy\n",
+ "import pprint\n",
+ "\n",
+ "from nerval import crm\n",
+ "from nerval import plot_confusion_matrix\n",
+ "from nervaluate import Evaluator"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "12cb2d3b-43be-4ac0-ba84-109917a5818d",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'0.12.2'"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# check version\n",
+ "print(spacy.__version__)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0cad211c",
+ "metadata": {},
+ "source": [
+ "## Installation mit pip \n",
+ "\n",
+ "Zur Installation von flair in eine Programmierumgebung kann der Paketmanager `pip` verwendet werden. \n",
+ "\n",
+ "Im JupyterHub der Humboldt-Universität zu Berlin steht Ihnen die flair-Bibliothek in der NLP-Umgebung bereits zur Verfügung. Sie müssen keine Installation vornehmen."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "338cca0d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# !pip install spacy\n",
+ "# !pip install nerval\n",
+ "# !pip install nervaluate"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72b6381b-7b8b-44d5-a714-e0387fbe5483",
+ "metadata": {},
+ "source": [
+ "## NER und Durchführung der Evaluation\n",
+ "\n",
+ "In den nächsten Schritten führen wir die NER mit dem spaCy-Medium-Modell durch und evaluieren die Ergebnisse mit der Python-Bibliothek nerval auf der Ground Truth.\n",
+ "\n",
+ "### Textdaten einlesen\n",
+ "\n",
+ "Der Text aus der Datei `example-news-text.txt`ist entnommen von:\n",
+ "\n",
+ "https://www.zdf.de/nachrichten/politik/bergkarabach-armenien-aserbaidschan-diplomatie-100.html"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "id": "6301e0b1-b7d5-4a38-a4ba-7a0c901d36f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "with open('example-news-text.txt') as f:\n",
+ " text = f.read()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "id": "8470b909-a1bb-481c-ba94-d09ef3a5f4a9",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Konflikt um Kaukasus-Region : Wie die Diplomatie bei Bergkarabach versagt\n",
+ "von Nina Niebergall, Eriwan\n",
+ "\n",
+ "Datum:\n",
+ "05.10.2023 14:29 Uhr\n",
+ "\n",
+ "Die Welt konnte oder wollte nicht helfen, als hunderttausend Armenier aus Bergkarabach vertrieben wurden. Der Konflikt ist nicht vorbei. Doch die Diplomatie hat keine L\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(text[:300])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c7f8de39-ac48-4a63-8e14-6c0f9feae6ad",
+ "metadata": {},
+ "source": [
+ "### Laden des Sprachmodells\n",
+ "\n",
+ "Das Sprachmodell von spaCy kann auch manuell in ein Verzeichnis geladen werden, siehe dazu das Notebook [download-spacy-models.ipynb](https://scm.cms.hu-berlin.de/digital-history/lehre/learning-bricks/-/blob/6bec726ea5329fd5f26a4127ebd93319b31e12c2/download_spacy_models/download-spacy-models.ipynb). Entsprechend dem Speicherort des Modells muss der Pfad in der nachfolgenden Codezelle angepasst werden. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "id": "90ea9960-6581-4cb5-97d3-195853f29b1f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Please uncomment the following line to download the model\n",
+ "# !python -m spacy download de_core_news_md"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "4a2c63d6-eccd-4b28-aefb-183e1509eec5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "nlp = spacy.load('de_core_news_md') "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a3a0f13e",
+ "metadata": {},
+ "source": [
+ "### Doc-Objekt erstellen"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1643a5a4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This is were the magic happens!\n",
+ "doc = nlp(text)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d67b9017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print('Anzahl der Token im Text:', doc.__len__())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fbf6dce2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print('Anzahl der Named Entities:', len(doc.ents))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f05f099f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print('Anzahl der Sätze:', sum(1 for _ in doc.sents))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d644882a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# check, show unique entities\n",
+ "for entity in ['PER', 'ORG', 'LOC', 'MISC']:\n",
+ " print(entity)\n",
+ " print(len({ ent.text for ent in doc.ents if ent.label_ == entity }))\n",
+ " print(sorted(list({ ent.text for ent in doc.ents if ent.label_ == entity })))\n",
+ " print('\\n')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ccf14038",
+ "metadata": {},
+ "source": [
+ "### Konvertieren der Ausgabe ins IOB-Format\n",
+ "\n",
+ "Die folgende List Comprehension wandelt die Ausgabe in das gewünschte IOB-Format um."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "902f1608",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "iob_format_tags = [ f'{token} {token.ent_iob_}-{token.ent_type_}' if token.ent_iob_ else 'O' for t in doc ]\n",
+ "\n",
+ "# from: https://prodi.gy/docs/named-entity-recognition#tip-offsets-biluo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7d389512",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "iob_format_tags[:15]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "89022673",
+ "metadata": {},
+ "source": [
+ "### Vorbereitung für die Evaluation\n",
+ "\n",
+ "Die Python Bibliothek nerval benötigt eine einfach geschachtelte Liste, um die Evaluation durchzuführen. Die Liste mit den mit den IOB-Tags wird mit der folgenden List-Comprehension in die entsprechende Struktur gebracht - wir benötigen hier nur den IOB-Tag, das Token selbst wird nicht gebraucht.\n",
+ "\n",
+ "Optional kann das Ergebnis der NER mit spaCy in einer txt-file zwischengespeichtert und dann wieder eingelesen werden."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "387b34ff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# with open('spacy-iob-format-tags.txt', 'w', encoding='utf8') as f:\n",
+ "# for line in iob_tags:\n",
+ "# line = line.replace(' O-', ' O') # whitespace in front of O \n",
+ "# if not line.startswith('\\n') and not line.startswith(' '): # filters newlines and lines with whitespace\n",
+ "# f.write(f'{line}\\n')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a232e4f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# with open('spacy-iob-format-tags.txt', 'r', encoding='utf8') as f:\n",
+ "# spacy_iob_pred = [[ line.rstrip().split()[1] for line in f ]] # nested list"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e1baabff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "spacy_iob_pred = [[ line.rstrip().split()[1] for line in iob_format_tags ]]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2930bde6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "spacy_iob_pred"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "09858474",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# sanity check\n",
+ "len(spacy_iob_pred[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "36fb3e33",
+ "metadata": {},
+ "source": [
+ "Dann kann die Ground Truth eingelesen werden. Hier sind noch einige newlines sowie whitespaces enthalten. Diese werden mit der List Comprehension herausgefiltert."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "id": "02adf5e7-6b4a-49e2-9bc5-02f55ed7028e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "with open('ground-truth-example-news-text.conll', 'r', encoding='utf8') as f:\n",
+ " iob_true = [[ line.rstrip().split()[1] for line in f if not line.startswith('\\n') and not line.startswith(' ')]] # nested list"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 51,
+ "id": "5514a409-115e-4529-9eb0-5319f2ff15a2",
+ "metadata": {
+ "collapsed": true,
+ "jupyter": {
+ "outputs_hidden": true
+ },
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[['O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'I-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'I-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'I-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'I-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'I-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'I-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-ORG',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-PER',\n",
+ " 'I-PER',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O',\n",
+ " 'B-LOC',\n",
+ " 'O']]"
+ ]
+ },
+ "execution_count": 51,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "iob_true"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "id": "04d072fc-ea40-4a46-8e32-32b17d3eedb7",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "979"
+ ]
+ },
+ "execution_count": 52,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# sanity check\n",
+ "len(iob_true[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "77bfd94a-459d-4961-8688-ba4e0a8b3140",
+ "metadata": {},
+ "source": [
+ "## Evaluation mit nerval\n",
+ "\n",
+ "Im letzten Schritt erfolgt die Evaluation mit nerval mittels nachfolgendem Codesnippet."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 53,
+ "id": "248b90af-ef14-43d4-98d0-5a2c818bf412",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "True Entities: 120\n",
+ "Pred Entities: 117 \n",
+ "\n",
+ "True Entities with 3 or more tags: 2\n",
+ "Pred Entities with 3 or more tags: 2 \n",
+ "\n",
+ "True positives: 112\n",
+ "False positives (true = 'O'): 0\n",
+ "False positives (true <> pred): 5\n",
+ "ToT False positives: 5\n",
+ "False negatives: 3 \n",
+ "\n",
+ " precision recall f1_score true_entities pred_entities\n",
+ "LOC 1.00 0.94 0.97 83.00 78.00\n",
+ "PER 0.93 1.00 0.96 26.00 28.00\n",
+ "ORG 1.00 0.73 0.84 11.00 8.00\n",
+ "LOC__ 0.00 0.00 0.00 0.00 1.00\n",
+ "MISC 0.00 0.00 0.00 0.00 2.00\n",
+ "micro_avg 0.96 0.93 0.95 120.00 117.00\n",
+ "macro_avg 0.59 0.53 0.55 120.00 117.00\n",
+ "weighted_avg 0.98 0.93 0.96 120.00 117.00\n"
+ ]
+ }
+ ],
+ "source": [
+ "cr, cm, cm_labels = crm(iob_true, spacy_iob_pred, scheme='BIO')\n",
+ "print(cr)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5c3cb689-1e25-4fa2-838c-155f59e8a9d2",
+ "metadata": {},
+ "source": [
+ "### Visualisierung mit nerval\n",
+ "\n",
+ "Die Ergebnisse der Evaluationen können mit der folgenden Codezelle in einer Heatmap visualisiert werden."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "id": "cf029642-56ff-49a9-b0c9-4ca108976f39",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAywAAAMcCAYAAACy9CEwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAABsBUlEQVR4nO3dd3RU1frG8ecMSSa0CS0FJIUSmoCgIEWuooAggqKCqBSR2C4odhS5Ct4fGOGKeJVmgyAqggXsgCggKAhIlSItCaFEikkGgQRC5vdHzFyHZCCh5GzI97PWWcs5Z+bMM7MF8ma/Zx/L4/F4BAAAAAAGctgdAAAAAAD8oWABAAAAYCwKFgAAAADGomABAAAAYCwKFgAAAADGomABAAAAYCwKFgAAAADGomABAAAAYKwAuwMAAAAAJVlmZqaOHTtmd4x8goKCFBwcbHcMChYAAADALpmZmQotXVp/2h2kABEREUpMTLS9aKFgAQAAAGxy7Ngx/SnpMUlOu8P8TZaksampOnbsGAULAAAAUNI5ZVbBYhIKFgAAAMBmlsxaDcuyO8DfmPS9AAAAAIAPChYAAAAAxqIlDAAAALCZJbPasEzKwgwLAAAAAGNRsAAAAAAwFi1hAAAAgM0cMmsmgSwAAAAAUAgULAAAAACMRUsYAAAAYDNawvwzKQsAAAAA+KBgAQAAAGAsWsIAAAAAm3HjSP+YYQEAAABgLAoWAAAAAMaiJQwAAACwGauE+WdSFgAAAADwQcECAAAAwFi0hAEAAAA2Y5Uw/5hhAQAAAGAsChYAAAAAxqIlDAAAALAZq4T5Z1IWAAAAAPBBwQIAAADAWLSEAQAAADajJcw/k7IAAAAAgA8KFgAAAADGoiUMAAAAsBk3jvSPGRYAAAAAxqJgAQAAAGAsWsIAAAAAm7FKmH8mZQEAAAAAHxQsAAAAAIxFSxgAAABgM0tmzSSwShgAAAAAFAIFCwAAAABj0RIGAAAA2IwbR/rHDAsAAAAAY1GwAAAAADAWLWEAAACAzbhxpH8mZQEAAAAAHxQsAAAAAIxFSxgAAABgM1YJ848ZFgAAAADGomABAAAAYCxawgAAAACbsUqYfyZlAQAAAAAfFCwAAAAAjEVLGAAAAGAzWsL8MykLAFz0MjIy9NBDDyk6OloBAQGyLEtJSUnn9T0ty1Lbtm3P63tczNq2bSvLMmmBTwAoWShYAFzUfvnlF8XFxSk2NlZly5ZV6dKlVatWLfXp00fffvttsed56qmnNH78eDVp0kTPPvushg0bpgoVKhR7DrskJSXJsixZlqVLLrlEJ06cKPB569ev9z6vXr16Z/We/fr1K5bCEABwftASBuCilJOToyeffFJjx45VQECArrvuOt10000KDAzUjh079NVXX+m9997Tv//9bz333HPFluvrr79W3bp19dlnnxXbe27atEllypQptvcrjICAAO3Zs0dz585V586d8x1/5513FBAQoOzsbBvS+Xr33Xd15MgRu2MAuMhx40j/KFgAXJT+9a9/aezYsWrSpIk+/vhj1apVy+f40aNHNW7cOB08eLBYc+3Zs0dXX311sb7n2c5QnA+tW7fW2rVrNXny5HwFy7Fjx/T++++rc+fO+vzzz21K+D9RUVF2RwCAEo2WMAAXnW3btmn06NGqXLmy5syZk69YkaTSpUvrqaee0gsvvOCz/+DBg3rsscdUo0YNOZ1OhYWFqWfPntq4cWO+c/y91WjChAmqX7++goODFR0drRdeeEE5OTn5nuvxeLRo0SJvu1O/fv3ynetkw4cPl2VZWrhwoc/+Tz75RNdcc43CwsIUHBysyMhIderUSbNnz/Z5nr9rWM7XZy2M0qVLq2fPnvriiy904MABn2Off/65Dhw4oHvuuafA1+7Zs0fDhg1Ty5YtFRYWJqfTqZiYGA0YMED79u3zeW5MTIymTp0qSapRo4b3e//795H3ePfu3erXr58iIiLkcDi83/fJ17BkZmaqUaNGCgwM1LJly3ze7+jRo2rQoIGCgoK0YsWKIn0nAICCMcMC4KKTkJCgEydO6IEHHlB4ePgpn+t0Or3/ffDgQbVs2VLbtm1T27ZtdccddygpKUkff/yxvvrqK3377bdq1apVvnM89dRTWrhwobp06aLrr79es2fP1vDhw3Xs2DGNHDlSktStWzfFxMTohRdeUHR0tLdQadKkyRl9xokTJ2rAgAGqWrWqbrnlFlWuXFl79+7V8uXLNXv2bHXr1u2Urz+fn7Ww+vfvrzfffFPvv/++HnnkEe/+yZMnKywsTF26dCnwdT/88IPGjBmjdu3aqUWLFgoMDNTq1as1ceJEzZ07V6tWrVJISIgk6dFHH1VCQoLWrl2rRx55xHu9UExMTL7vo1WrVqpUqZJ69uypY8eOyeVyFfj+wcHBmj59upo3b6677rpLa9as8T73scce06ZNmxQfH6/mzZsX6fsAULKxStgpeADgItO2bVuPJM/8+fOL9Lr+/ft7JHmGDBnis3/OnDkeSZ7Y2FjPiRMnvPvvvvtujyRPjRo1PHv27PHu379/v6dChQqe8uXLe7KysnzOJclzzTXX5HvvvHMlJibmOzZs2DCPJM+CBQu8+y6//HJPUFCQZ9++ffmef+DAgdO+Z3F81oIkJiZ6JHk6duzo8Xg8nksvvdTTuHFj7/Fdu3Z5SpUq5XniiSe82evWretzjt9//91z6NChfOeeOnWqR5JnxIgRPvtP9d3mvYckzz333OPJzs7Od/yaa67xFPTP5bhx4zySPHfddZfH4/F4Zs+e7ZHkufbaa32+OwA4lYyMDI8kz3jJM9mgbfxffzdmZGTY/RV5jCqeAOBcSE1NlSRVr1690K85duyYpk+frsqVK+tf//qXz7GOHTuqY8eO2rp1q3766ad8r33uuedUtWpV7+MqVaro5ptv1qFDh/Tbb7+d4ac4vcDAQAUGBubbX7ly5VO+zqTPes8992jdunX65ZdfJP1vdqx///5+XxMWFqZy5crl29+nTx+5XC7Nnz+/yDmCgoI0evRolSpVqtCvGThwoLp27aoPPvhAo0ePVlxcnCpVqqR3331XDgf/vALAucLfqAAgafPmzTp69KiuvPLKAlfUyrvmYc2aNfmOXX755fn25RVL6enp5zKm1+23367Dhw+rYcOGevLJJ/Xll18W+r1M+qx9+vRRYGCgJk+eLCm3YGnRooUaNGhwytd9+umn6tixo0JDQ733s3E4HHK73dqzZ0+Rc9SoUUNVqlQp8usmT56sqlWr6umnn9bBgwf11ltvFalQBoA8DgM3U5iUBQDOiYiICEnS7t27C/0at9stSX6veck7Z0ZGRr5jeddL/F1AQO4lgv7uM3K2Bg8erLfeeksRERF65ZVX1LVrV4WGhurmm29WYmLiKV9r0mcNCwtT586dNX36dM2dO1fbtm3ze7F9njFjxui2227T6tWrdf311+uJJ57QsGHDNGzYMIWEhCgrK6vIOU53rZM/VapU0T/+8Q9JUnR0tG666aYzOg8AwD8KFgAXnauuukqS9N133xX6NXkXTf/+++8FHs/b7+9C7LOV10JU0H1HCiocLMvSvffeq5UrV2r//v2aNWuWbr31Vn3++ee68cYbT1k82P1ZT9a/f3+lpaUpLi5OpUuX1p133un3udnZ2fq///s/VatWTRs2bND777+vUaNGafjw4Ro2bJiOHTt2RhnO9E72H330kWbOnKnKlSsrOTlZw4YNO6PzAAD8o2ABcNHp16+fSpUqpTfffFP79+8/5XPzfhtfr149BQcHa8WKFQXeJHDRokWSznxVr9OpWLGipIJnhVavXn3K11auXFndunXTjBkzdN1112nTpk3atm2b3+fb/VlP1rlzZ0VERGj37t267bbbTlkoHThwQBkZGWrZsqVCQ0N9jq1cuVJHjx7N95q861LO9WzXzp07df/99yssLExr1qxRs2bN9NJLL3m/PwAoCsvAzRQULAAuOrVr19bgwYN14MAB3XDDDQW2SGVmZuqVV17R8OHDJeVedH3nnXfqwIEDio+P93nu/Pnz9c0336h27dre2ZtzrVmzZpJyr+H4u48//rjAH4Dnzp2bbzbm+PHj+uOPPyTl3ufEH7s/68kCAgL0+eefa9asWaddGjksLEylS5fWqlWrfIqttLQ0PfzwwwW+plKlSpKkXbt2nbPMOTk56t27t9LT05WQkKDq1avrgw8+UOnSpdWnTx+lpaWds/cCgJKO+7AAuCiNGDFCmZmZGjt2rOrWravrrrtODRs2VGBgoBITEzV//nwdPHhQI0aM8L5m1KhRWrRokUaMGKGffvpJLVq08N6bpEyZMpoyZcp5W/2pW7duqlGjhhISEpSSkqKmTZtq06ZN+v7779W5c2d9/fXXPs/v2bOnypQpozZt2ig6OlrHjx/Xt99+q40bN6pnz56nvTu7nZ+1IM2bNy/UfUscDocGDBigMWPG6LLLLlPXrl3ldrv1zTffKDo6WtWqVcv3muuuu04vv/yyHnjgAfXo0UNly5ZVVFSU7rrrrjPOO3LkSC1evFiDBg3SDTfcIEmKjY3Va6+9pri4ON1///366KOPzvj8AID/YYYFwEXJ4XDolVde0YoVK9SnTx9t375dEyZM0NixY/Xzzz/r+uuv17x58zR06FDva0JDQ/Xzzz9r0KBB2r59u15++WV9++23uvnmm/Xzzz+rTZs25y1v6dKl9d133+nmm2/W8uXLNXHiRGVmZuqHH34o8Af5vBsTLl++XOPGjdN7772n8uXL64033tB777132vez87Oerfj4eI0cOVKWZWnChAn69ttvdccdd2jevHkFLvN8ww03aPTo0crJydGoUaM0ZMgQvfnmm2f8/suWLdO///1vNWzYUKNGjfI51r9/f3Xv3l0ff/yx3n777TN+DwAlj90rgpm8Spjl8Xg8docAAAAASiK3262QkBC9Kcl/M2/xOyrpfuUu/FJci7D4Y1LxBAAAAAA+uIYFAAAAsJkls2YSWCUMAAAAAAqBggUAAACAsWgJAwAAAGxm2s0aTcrCDAsAAAAAYzHDcpZycnK0Z88elS9fXpZlUi0KAAAASfJ4PDp06JCqVatWrDfFxblBwXKW9uzZo8jISLtjAAAA4DRSUlJUvXp1u2MUyLSbNRY1S3p6utq2bet9fOTIEe3YsUP79u1Tdna2+vbtq+3bt8vpdGrSpElFukExBctZKl++vCTpMUlOe6OgAM/s3Wp3BJyC5eCvIFN5jh+2OwL8sALL2h0Bp+DJybY7AgrgPnRIUXUu9/7chnOvQoUKWrNmjffxyy+/rEWLFqlSpUrq37+/WrZsqTlz5mjFihXq3r27tm/froCAwv0cwE8LZymvDcwpChYTuVz8xWQyChZzeY6b9Hs+/B0Fi9koWMxG+37xmTJlikaOHClJmjlzphITEyVJzZs3V3h4uJYsWeIzI3Mq/LQAAAAA2MzUVcLcbrfPfqfTKafz1L+mX7p0qQ4ePKguXbro4MGDysnJUWhoqPd4TEyMdu7cWegs/AoNAAAAQIEiIyMVEhLi3eLj40/7msmTJ6tv377elq+TZ7Y8Hk+RMjDDAgAAAKBAKSkpcrlc3senm105fPiwZsyYoeXLl0uSKleuLEnav3+/d5YlOTlZUVFRhc7ADAsAAABgM4eBmyS5XC6f7XQFy0cffaTGjRurXr163n09evTQ+PHjJUkrVqxQamoqq4QBAAAAKH7vvPOO4uLifPaNGjVKffr0UWxsrIKCgjRt2rRCrxAmUbAAAAAAOEcWL16cb194eLjmzZt3xuekYAEAAABsdqHfOPJ8MikLAAAAAPigYAEAAABgLFrCAAAAAJuZeuNIEzDDAgAAAMBYFCwAAAAAjEVLGAAAAGAzVgnzz6QsAAAAAOCDggUAAACAsWgJAwAAAGxGS5h/JmUBAAAAAB8ULAAAAACMRUsYAAAAYDNuHOkfMywAAAAAjEXBAgAAAMBYtIQBAAAANrMsybLMacSyPB7JY3eKXMywAAAAADAWBQsAAAAAY9ESBgAAANjMsiyzWsIkyWNGTxgzLAAAAACMRcECAAAAwFi0hAEAAAA2sxyWHCa1hHkk5dASBgAAAACnRMECAAAAwFi0hAEAAAA2M3KVMEMwwwIAAADAWBQsAAAAAIxFSxgAAABgM8thWEuYGQuESWKGBQAAAIDBKFgAAAAAGIuWMAAAAMBmrBLmHzMsAAAAAIxFwQIAAADAWLSEAQAAADZjlTD/mGEBAAAAYCwKFgAAAADGoiUMAAAAsBmrhPnHDAsAAAAAY12QBUu/fv3UrVu3Ao8dPXpUw4YNU926deV0OlWlShV1795dGzZsyPdct9utoUOHql69egoODlZERITat2+vTz/9VB6PQVcanSfBIS49uGyBd3t43c96/lCqSlesoGqXN1Hcwm/0wNLvNXD1T7rq8YftjluiffPkUL1av5leKBuhfRs22R0HJzm4bbvebnuDXmt0pd5s00H7Nv1mdyT8ZdrNvTWxVUdNuuoGTenYXanr8v9bAHvw58Zc/JsD01xULWFZWVlq3769du7cqTFjxqhFixb6/fffFR8frxYtWmj+/Plq2bKlJCk9PV1t2rRRRkaGRowYoebNmysgIECLFi3S4MGDdd1116lChQr2fqDzLDPDrUktr/U+bv3oQEW3aa2jaenq+/UrWjhitH77ao5KV6ygh9Ys1Zav52r/5i02Ji65GnTroqseG6jJ7W+yOwoK8MVDT+iKuL5q2udObfj0c33+4CO6d9Ecu2NBUo+p4xVcIUSStPnLufps4FN6YPHXNqeCxJ8bk/Fvjj1YJcy/i6pgefXVV7V06VKtXr1al112mSQpOjpan3zyiVq0aKG4uDj9+uuvsixLzz77rJKSkrRlyxZVq1bNe446derozjvvVHBwsF0fwzZN+typ74eP9D4ODnFJkgLLltGJ48d1NC3dpmSIbtPK7gjw4899+7V3zTr1+fJjSVKDW7rq68efUVryTlWMjrI5HfKKFUnKzDgky3FBNhZcdPhzYzb+zYFpLqqC5YMPPlCHDh28xUoeh8Ohxx57TL169dLatWvVuHFjffjhh+rVq5dPsZKnXLlyxRXZGNVbNFOZShW15et5kqTPHhikOz6apuuGP6syVSrri4GP68/f99mcEjCPe9dula8aoVIBuX+dWpalkOqXKCNlFz94GWLW/Y8pafFSSVKvT9+1OQ0k/twAKJqLqmDZsmWLrr322gKP1a9f3/ucatWqKS0tTfXq1Svye2RlZSkrK8v72O12n1lYwzTt20trP5ipnBMnJEmtH3tI3z47XBs++UwVY6LVb95n2r1ytQ5u3WZzUsA8+abwS8A1cBeSW94cK0la8/7H+vZfI9Xrk6k2J4LEnxvgZKwS5l+JmRvPu4jesiyf/y6q+Ph4hYSEeLfIyMhzmtMOgWXKqGH3blo99X1JUpnKlVT/ps7a8MlnkqS0pGTtWvGLIls2tzMmYCRX9Uvk3r1HJ7KzJeX+XZOxe49CIqvbnAwna9Kru5IWL9WRg2l2Rynx+HMDoCguqoKlTp062rhxY4HHNm/eLEmKjY1VaGioKlasqE2bir7yxZAhQ5SRkeHdUlJSziqzCS699Sb9/usGHdiSO3tyNC1d2ZlZim7TWlJuAVP9ymbat3GznTEBI5ULC1XEZY20bvpHkqSNs75QhahI2loMkOU+pEN7f/c+3vTFHJWuVFGlK1WwLxQk8ecGQNFcVC1hd9xxh4YOHaq1a9f6XMeSk5OjsWPHqkGDBrrssstkWZZ69uypadOmadiwYfmuYzl8+LCcTqcCAvJ/PU6nU06n87x/luLUtF8vrUp43/vYk5Ojj3rHqeOo/5MjoJRKBQbqp1fHa88vq21MWbJ99dgz+u3Lufrz9316t8vtCipXVoPWL7M7Fv7SddwYzb7vYS0ePVZOV3nd8tZ4uyNBUqb7kGb2flDZmZmyHA6VqVJJd82cbFTLRUnGnxtz8W+OPRwOSw6D/n5yGNSlaXkuwBuO9OvXT8nJyRo7dqzP/kqVKun222/Xnj17fJY1fvHFF/Xtt9/6LGuclpam1q1b688//9TIkSPVrFkzBQYGavHixYqPj9eKFSsKtayx2+1WSEiInpF0cZUxF4dhh1PtjoBTsBwX1e9MLiqe44ftjgA/rMCydkfAKXhysu2OgAK43YdUoWqsMjIy5HK57I7jI+9nya8qlFFZgwqWwx6Pbkw/YsR3dsH+tLBw4UI1bdrUZ9/dd9+t77//XvHx8Xr22WeVnJys8uXL69prr9WyZcvUsGFD73MrVqyoZcuW6aWXXtKIESOUnJysihUrqlGjRvrPf/6jkJCQk98SAAAAQDG7IGdYTMIMi9mYYTEbMyzmYobFXMywmI0ZFjNdCDMsX1csa9wMS+e0w0Z8ZxfVRfcAAAAALi4ULAAAAACMRT8GAAAAYDPLsmQ5zGkJs3LsTvA/zLAAAAAAMBYFCwAAAABj0RIGAAAA2MyyLKNubGtQFGZYAAAAAJiLggUAAACAsWgJAwAAAGxmOQxbJczuAH/DDAsAAAAAY1GwAAAAADAWLWEAAACAzVglzD9mWAAAAAAYi4IFAAAAgLFoCQMAAABsxiph/jHDAgAAAMBYFCwAAAAAjEVLGAAAAGAzVgnzjxkWAAAAAMaiYAEAAABgLFrCAAAAAJtZDslh1CphHrsjeDHDAgAAAMBYFCwAAAAAjEVLGAAAAGAz81YJMycLMywAAAAAjEXBAgAAAMBYtIQBAAAANrMcliyjVgkzJwszLAAAAACMRcECAAAAwFi0hAEAAAA2Y5Uw/5hhAQAAAGAsChYAAAAAxqIlDAAAALCZ5cjdTGFOQxgzLAAAAAAMRsECAAAAwFi0hAEAAAA2Y5Uw/5hhAQAAAGAsChYAAAAAxqIlDAAAALCZZVmyHOa0YVkec7IwwwIAAADAWBQsAAAAAIxFwQIAAADYzGFZxm1FlZWVpYceekixsbG69NJL1bt3b0nSvn371KlTJ8XGxqphw4ZasmRJkc7LNSwAAAAAztozzzwjh8OhLVu2yLIs7d2717u/ZcuWmjNnjlasWKHu3btr+/btCggoXClCwQIAAADgrBw+fFhTpkzRrl27vPdwqVq1qiRp5syZSkxMlCQ1b95c4eHhWrJkidq2bVuoc1OwAAAAADazHGauEuZ2u332O51OOZ3OfM/fvn27KleurBEjRmj+/PkqXbq0hg8friZNmignJ0ehoaHe58bExGjnzp2FzkLBco4M+T1RLld5u2MAwDlhBZa1OwJwQbIc/GhlIsblzEVGRvo8HjZsmIYPH57vecePH9eOHTvUoEEDvfTSS1q7dq3at2+vX3/91Tvjksfj8RQpA6MHAAAAoEApKSlyuVzexwXNrkhSdHS0HA6HevXqJUm67LLLVKNGDW3atEmStH//fu8sS3JysqKiogqdgVXCAAAAAJtZlmXcJkkul8tn81ewVKlSRe3atdPcuXMl5RYliYmJqlu3rnr06KHx48dLklasWKHU1FS1adOm0N8NMywAAAAAztqkSZPUv39/Pf300ypVqpTefPNNVa1aVaNGjVKfPn0UGxuroKAgTZs2rdArhEkULAAAAADOgZo1a2rhwoX59oeHh2vevHlnfF4KFgAAAMBmpq4SZgKuYQEAAABgLAoWAAAAAMaiJQwAAACw2d9X5jKBSVmYYQEAAABgLAoWAAAAAMaiJQwAAACwGauE+ccMCwAAAABjUbAAAAAAMBYFCwAAAABjcQ0LAAAAYDOWNfaPGRYAAAAAxqJgAQAAAGAsWsIAAAAAm1kOhyyHOXMJlsfuBP9jzrcCAAAAACehYAEAAABgLFrCAAAAAJtxp3v/mGEBAAAAYCwKFgAAAADGoiUMAAAAsJtl5W6mMCgLMywAAAAAjEXBAgAAAMBYtIQBAAAANrMsw1YJyzEnCzMsAAAAAIxFwQIAAADAWLSEAQAAAHZzWLIcBs0lODx2J/Ay6FsBAAAAAF8ULAAAAACMRUsYAAAAYDPLsmQZdLNGk7IwwwIAAADAWBQsAAAAAIxFSxgAAABgN4eVu5nCoCzMsAAAAAAwFgULAAAAAGPREgYAAADYzHI4jLpxpMWNIwEAAADg9ChYAAAAABiLljAAAADAZtw40j9mWAAAAAAYi4IFAAAAgLFoCQMAAABsZjksWQbdrNGkLMywAAAAADAWMyzwOrhtu2bd+5COHDyo4JAQdXtrnMLq17U7FsTYmI7xMRdjYy7GxmyMD0xi6wxLv3791K1btwKPHT16VMOGDVPdunXldDpVpUoVde/eXRs2bMj3XLfbraFDh6pevXoKDg5WRESE2rdvr08//VQejzk3vTHdFw89oSvi+mrQ+uW66vGH9fmDj9gdCX9hbMzG+JiLsTEXY2M2xscGDsu8zRBGtoRlZWWpffv2mjx5sv7v//5PW7Zs0ddff60TJ06oRYsWWrZsmfe56enpat26td59910NGTJEq1at0g8//KCePXtq8ODBysjIsPGTXDj+3Ldfe9esU+M7e0iSGtzSVWnJO5WWvNPmZGBszMb4mIuxMRdjYzbGB6YxsiXs1Vdf1dKlS7V69WpddtllkqTo6Gh98sknatGiheLi4vTrr7/Ksiw9++yzSkpK0pYtW1StWjXvOerUqaM777xTwcHBdn2MC4p7126VrxqhUgG5/0tYlqWQ6pcoI2WXKkZH2ZyuZGNszMb4mIuxMRdjYzbGB6Yxcoblgw8+UIcOHbzFSh6Hw6HHHntMGzdu1Nq1a5WTk6MPP/xQvXr18ilW8pQrV04BAee2JsvKypLb7fbZLhb5bhBEO50xGBuzMT7mYmzMxdiYjfEpfpblMG4zhTlJ/mbLli2qX79+gcfy9m/ZskUHDhxQWlqa6tWrV2zZ4uPjFRIS4t0iIyOL7b3PJ1f1S+TevUcnsrMlSR6PRxm79ygksrrNycDYmI3xMRdjYy7GxmyMD0xjZMFyKnkX0VuW5fPfxWXIkCHKyMjwbikpKcX23udTubBQRVzWSOumfyRJ2jjrC1WIimTq1wCMjdkYH3MxNuZibMzG+MA0Rl7DUqdOHW3cuLHAY5s3b5YkxcbGKjQ0VBUrVtSmTZuKLZvT6ZTT6Sy29ytOXceN0ez7Htbi0WPldJXXLW+NtzsS/sLYmI3xMRdjYy7GxmyMT/HjxpH+WR4b1/3t16+f0tPTNXv2bJ/98fHxGjp0qM9F95KUk5OjFi1a6MiRI96L7v/5z39q2rRp+S66l6TDhw/L6XSe8+tY/s7tdiskJEQZvyfK5Sp/3t4HAAAAZ8btPqSQ8BrKyMiQy+WyO46PvJ8lf72ukcoHlLI7jteh7BNq+P16I74z21vCMjIytGbNGp+tV69euvLKK9W1a1d99NFH2rlzp1asWKHbbrtNmzZt0jvvvONtA3vxxRcVGRmpFi1a6N1339XGjRu1detWTZ48WU2aNNGff/5p8ycEAAAAcKZsbwlbuHChmjZt6rPv7rvv1vfff6/4+Hg9++yzSk5OVvny5XXttddq2bJlatiwofe5FStW1LJly/TSSy9pxIgRSk5OVsWKFdWoUSP95z//UUhISHF/JAAAAKBIaAnzz9aWsIsBLWEAAABmuxBawja0b2xcS9il89cZ8Z3Z3hIGAAAAAP7Y3hJ2vi1evFg33HBDgceOHj2q0qVL+30t178AAACgeDgkg27WKJnThHXRFyzNmjXTmjVrCjx2uoIFAAAAgL0u+oKldOnSql27tt0xAAAAAJyBi75gAQAAAEzHKmH+mdQoBwAAAAA+KFgAAAAAGIuWMAAAAMBmtIT5xwwLAAAAAGNRsAAAAAAwFi1hAAAAgM0sy5JlmdOGZVIWZlgAAAAAGIuCBQAAAICxaAkDAAAA7OawJIdBcwmOHLsTeBn0rQAAAACALwoWAAAAAMaiJQwAAACwGTeO9I8ZFgAAAADGomABAAAAYCxawgAAAACbceNI/5hhAQAAAGAsChYAAAAAxqIlDAAAALCZ5XDIMujGkSZlMScJAAAAAJyEggUAAACAsWgJAwAAAGzGjSP9Y4YFAAAAgLEoWAAAAAAYi5YwAAAAwG6WlbuZwqAszLAAAAAAMBYFCwAAAABj0RIGAAAA2IxVwvxjhgUAAACAsShYAAAAABiLljAAAADAZpblkOUwZy7BsszJYk4SAAAAABesmJgY1atXT02aNFGTJk00Y8YMSdK+ffvUqVMnxcbGqmHDhlqyZEmRzssMCwAAAIBz4uOPP1bDhg199j3zzDNq2bKl5syZoxUrVqh79+7avn27AgIKV4pQsAAAAAA2syxLlkE3azyXWWbOnKnExERJUvPmzRUeHq4lS5aobdu2hXo9BQsAAACAArndbp/HTqdTTqfT7/N79eqlnJwctWjRQvHx8XI4HMrJyVFoaKj3OTExMdq5c2ehM3ANCwAAAIACRUZGKiQkxLvFx8f7fe4PP/ygtWvXatWqVapcubLuvvtuSflnazweT5EyMMMCAAAA2M1h5W6m+CtLSkqKXC6Xd/epZleioqIkSYGBgXr00UdVp04dVa5cWZK0f/9+7yxLcnKy97mFilLk8AAAAABKBJfL5bP5K1gOHz6s9PR07+Pp06eradOmkqQePXpo/PjxkqQVK1YoNTVVbdq0KXQGZlgAAAAAnJXff/9dt912m06cOCGPx6OaNWvq3XfflSSNGjVKffr0UWxsrIKCgjRt2rRCrxAmUbAAAAAA9nOYdeNIFTFLzZo1tXr16gKPhYeHa968eWce5YxfCQAAAADnGQULAAAAAGPREgYAAADY7GK+ceTZYoYFAAAAgLEoWAAAAAAYi5YwAAAAwGaWw5Jl0I0jTcrCDAsAAAAAY1GwAAAAADAWLWEAAACA3SwrdzOFQVmYYQEAAABgLAoWAAAAAMaiJQwAAACwmeUwa2Uuy6BpDYOiAAAAAIAvChYAAAAAxqIlDAAAALCb9ddmCoOyMMMCAAAAwFgULAAAAACMRUsYAAAAYDduHOkXMywAAAAAjEXBAgAAAMBYtIQBAAAANqMjzD9mWAAAAAAYi4IFAAAAgLFoCQMAAADs5rByN1MYlIUZFgAAAADGomABAAAAYCxawgAAAACbsUqYf8ywAAAAADAWBQsAAAAAY9ESBgAAANjOsJ4wmZOFGRYAAAAAxqJgAQAAAGAsWsIAAAAAuzlk1lSCQVkMigIAAAAAvihYAAAAABiLljAAAADAZpZlyTJolTCTsjDDAgAAAMBYFCwAAAAAjEVLGAAAAGA3y7AbRxqUhRkWAAAAAMaiYAEAAABgLFrCAAAAAJvREeYfMywAAAAAjEXBAgAAAMBYtIQBAAAAdnNYuZspDMrCDAsAAAAAY1GwAAAAADAWLWHwOrhtu2bd+5COHDyo4JAQdXtrnMLq17U7FsTYmI7xMRdjYy7GxmyMjw2svzZTGJTF2BmWfv36ybIsPfjgg/mODRgwQJZlqV+/ft7nduvWzXt83759euCBBxQVFSWn06mIiAh17NhRS5cu9TnP6tWr1aNHD4WHhys4OFh16tTRfffdpy1btpzPj2asLx56QlfE9dWg9ct11eMP6/MHH7E7Ev7C2JiN8TEXY2MuxsZsjA9MYmzBIkmRkZH68MMPdfToUe++zMxMTZ8+XVFRUX5fd9ttt2nt2rWaOnWqtmzZos8//1xt27bVH3/84X3Ol19+qZYtWyorK0vvv/++Nm3apGnTpikkJETPPffcef1cJvpz337tXbNOje/sIUlqcEtXpSXvVFryTpuTgbExG+NjLsbGXIyN2RgfmMbolrDLL79cO3bs0KeffqpevXpJkj799FNFRkaqZs2aBb4mPT1dS5Ys0cKFC3XNNddIkqKjo3XllVd6n3PkyBHdc8896ty5s2bNmuXdX6NGDbVo0ULp6enn70MZyr1rt8pXjVCpgNz/JSzLUkj1S5SRsksVo/0Xhzj/GBuzMT7mYmzMxdiYjfGxh2VZsgy6W6NJWYyeYZGke+65R1OmTPE+njx5svr37+/3+eXKlVO5cuU0e/ZsZWVlFficuXPn6sCBAxo8eHCBxytUqOD3/FlZWXK73T7bxSLf/5gejz1BkA9jYzbGx1yMjbkYG7MxPjCJ8QVLnz59tGTJEiUlJSk5OVk//vijevfu7ff5AQEBSkhI0NSpU1WhQgVdddVVevbZZ7Vu3Trvc7Zu3SpJqlevXpHzxMfHKyQkxLtFRkYW/UMZyFX9Erl379GJ7GxJksfjUcbuPQqJrG5zMjA2ZmN8zMXYmIuxMRvjA9MYX7BUqVJFN954o6ZOnaopU6boxhtvVJUqVU75mttuu0179uzR559/ro4dO2rhwoW6/PLLlZCQICn3D96ZGjJkiDIyMrxbSkrKGZ/LJOXCQhVxWSOtm/6RJGnjrC9UISqSqV8DMDZmY3zMxdiYi7ExG+NjE8vAzRBGX8OSp3///nrooYckSePHjy/Ua4KDg9WhQwd16NBBzz//vO69914NGzZM/fr1U506dSRJmzdvVqtWrYqUxel0yul0Fu0DXCC6jhuj2fc9rMWjx8rpKq9b3ircd43zj7ExG+NjLsbGXIyN2RgfmOSCKFg6deqkY8eOSZI6dux4Rudo0KCBZs+eLUm6/vrrVaVKFY0ePdrnovs86enpp7yO5WJVpU6s7l00x+4YKABjYzbGx1yMjbkYG7MxPjDJBVGwlCpVSps2bfL+96kcPHhQPXr0UP/+/dW4cWOVL19eK1eu1OjRo3XzzTdLksqWLau3335bPXr00E033aRBgwapdu3aOnDggGbOnKmdO3fqww8/PO+fCwAAAJAky2HJcpjTh2VSlguiYJEkl8tVqOeVK1dOLVq00NixY7V9+3YdP35ckZGRuu+++/Tss896n3fzzTfrp59+Unx8vO666y653W5FRkbquuuu04gRI87XxwAAAABQBJbnbK5Ah9xut0JCQpTxe6JcrvJ2xwEAAMBJ3O5DCgmvoYyMjEL/Ery45P0smfpEF7mcgXbH8XJnHVfEmC+N+M4umBkWAAAA4KJl2MpcJmUxflljAAAAACUXBQsAAAAAY9ESBgAAANjNsnI3UxiUhRkWAAAAAMaiYAEAAABgLFrCAAAAAJvREeYfMywAAAAAjEXBAgAAAMBYtIQBAAAAdnNYuZspDMrCDAsAAAAAY1GwAAAAADAWLWEAAACAzSyZtTKXQVEKV7C89tprhT7hoEGDzjgMAAAAAPxdoQqWsWPHFupklmVRsAAAAAA4ZwpVsCQmJp7vHAAAAEDJxZ0j/Trji+6PHTum3377TdnZ2ecyDwAAAAB4FblgOXLkiOLi4lSmTBldeuml2rlzp6Tca1deeumlcx4QAAAAQMlV5IJlyJAhWrt2rRYuXKjg4GDv/vbt22vGjBnnNBwAAABQEuR1hJm0maLIyxrPnj1bM2bMUMuWLWX97ZM0aNBA27dvP6fhAAAAAJRsRZ5h2b9/v8LCwvLtP3z4sE8BAwAAAABnq8gFS/PmzfXVV195H+cVKW+99ZZatWp17pIBAAAAJYXDMm8zRJFbwuLj49WpUydt3LhR2dnZ+u9//6sNGzZo6dKlWrRo0fnICAAAAKCEKvIMS+vWrfXjjz/qyJEjqlWrlubNm6fw8HAtXbpUV1xxxfnICAAAAKCEKvIMiyQ1atRIU6dOPddZAAAAgJLJtKW5DMpyRgXLiRMnNGvWLG3atEmWZal+/fq6+eabFRBwRqcDAAAAgAIVucL49ddfdfPNNys1NVV169aVJG3ZskWhoaH6/PPP1ahRo3MeEgAAAEDJVORrWO69915deuml2rVrl1atWqVVq1YpJSVFjRs31v33338+MgIAAAAXNbtvEnlR3Thy7dq1WrlypSpWrOjdV7FiRY0cOVLNmzc/p+EAAAAAlGxFnmGpW7eufv/993z79+3bp9q1a5+TUAAAAAAgFXKGxe12e//7xRdf1KBBgzR8+HC1bNlSkrRs2TL9+9//1qhRo85PSgAAAOBiZloflkFZClWwVKhQwXtHe0nyeDy6/fbbvfs8Ho8kqWvXrjpx4sR5iAkAAADgQvDCCy9o+PDhWr9+vRo2bKh9+/apb9++2r59u5xOpyZNmqQ2bdoU+nyFKlgWLFhwxoEBAAAAlAyrVq3SsmXLFBUV5d33zDPPqGXLlpozZ45WrFih7t27a/v27YW+JUqhnnXNNdecWWIAAAAAp2U5cjdTnEmWrKwsDRw4UB988IGuvfZa7/6ZM2cqMTFRktS8eXOFh4dryZIlatu2baHOe8Z3ejxy5Ih27typY8eO+exv3LjxmZ4SAAAAgEH+fi27JDmdTjmdzgKf+/zzz6t3796qUaOGd9/BgweVk5Oj0NBQ776YmBjt3Lmz0BmKXLDs379f99xzj7755psCj3MNCwAAAHBxiIyM9Hk8bNgwDR8+PN/zli5dqhUrVuill17Kd8w66QL+vOvfC6vIBcujjz6qtLQ0LVu2TNdee61mzZql33//XSNGjNCYMWOKejoAAAAAhq4SlpKSIpfL5d3tb3Zl0aJF2rx5s3d2ZdeuXerYsaPefvttSbmTHnmzLMnJyT7XuJxOkbvTvv/+e40dO1bNmzeXw+FQdHS0evfurdGjRys+Pr6opwMAAABgKJfL5bP5K1ieeeYZ7dmzR0lJSUpKSlL16tU1d+5c3XDDDerRo4fGjx8vSVqxYoVSU1PP/Sphf3f48GGFhYVJkipVqqT9+/erTp06atSokVatWlXU0wEAAAC4iI0aNUp9+vRRbGysgoKCNG3atEKvECadQcFSt25d/fbbb4qJiVGTJk30xhtvKCYmRpMmTVLVqlWLejoAAAAA1l+bKc4yS1JSkve/w8PDNW/evDM+1xldw7J3715JuRfddOzYUe+//76CgoKUkJBwxkEAAAAA4GRFLlh69erl/e+mTZsqKSlJmzdvVlRUlKpUqXJOwwEAAAAo2c74Pix5ypQpo8svv/xcZAEAAABKJMuy8i3/ayeTshSqYHn88ccLfcJXXnnljMMAAAAAwN8VqmBZvXp1oU5mUiUGAAAA4MJXqIJlwYIF5zsHAAAAUHI5rNzNFAZlKfKNIwEAAACguFCwAAAAADDWWa8SBgAAAOAsWVbuZgqDsjDDAgAAAMBYFCwAAAAAjHVGBcu0adN01VVXqVq1akpOTpYkvfrqq/rss8/OaTgAAACgRHDofyuFGbHZ/YX8T5GjTJw4UY8//rg6d+6s9PR0nThxQpJUoUIFvfrqq+c6HwAAAIASrMgFy+uvv6633npLQ4cOValSpbz7mzVrpvXr15/TcAAAAABKtiKvEpaYmKimTZvm2+90OnX48OFzEgoAAAAoWRySZVAflkE9YUVOUqNGDa1Zsybf/m+++UYNGjQ4F5kAAAAAQNIZzLA89dRTGjhwoDIzM+XxeLR8+XJNnz5d8fHxevvtt89HRgAAAAAlVJELlnvuuUfZ2dkaPHiwjhw5orvuukuXXHKJ/vvf/+qOO+44HxkBAACAixs3jvTrjO50f9999+m+++7TgQMHlJOTo7CwsHOdCwAAAADOrGDJU6VKlXOVAwAAAADyKXLBUqNGDVmnmCLasWPHWQUCAAAASpy8GzaawqAsRS5YHn30UZ/Hx48f1+rVqzVnzhw99dRT5yoXAAAAABS9YHnkkUcK3D9+/HitXLnyrAMBAAAAQJ5zdkeYG264QZ988sm5Oh0AAABQclgO8zZDnLMkH3/8sSpVqnSuTgcAAAAARW8Ja9q0qc9F9x6PR6mpqdq/f78mTJhwTsMBAAAAKNmKXLB069bN57HD4VBoaKjatm2revXqnatcAAAAQMnBKmF+Falgyc7OVkxMjDp27KiIiIjzlQkAAAAAJBXxGpaAgAD985//VFZW1vnKAwAAAABeRb7ovkWLFlq9evX5yAIAAACUTJZl3maIIl/DMmDAAD3xxBPatWuXrrjiCpUtW9bneOPGjc9ZOAAAAAAlW6ELlv79++vVV19Vz549JUmDBg3yHrMsSx6PR5Zl6cSJE+c+JQAAAIASqdAFy9SpU/XSSy8pMTHxfOYBAAAASh6HI3czhUFZCl2weDweSVJ0dPR5CwMAAAAAf1ek0sky6OIbAAAAABe/Il10X6dOndMWLX/88cdZBQIAAABKHMNW5jIpS5EKlhdeeEEhISHnKwsAAAAA+ChSwXLHHXcoLCzsfGUBAAAAAB+FLli4fgUAAAA4T1glzK9CJ8lbJQwAAAAAikuhZ1hycnLOZw4AAAAAyKdI17AAAAAAOA9YJcwvc5rTAAAAAOAkFCwAAAAAjEVLGAAAAGA3WsL8YoYFAAAAgLEoWAAAAAAYi5YwAAAAwG7cONIvc5IAAAAAwEkoWAAAAAAYi5YwAAAAwG6WjFqZSwZFYYYFAAAAgLFKdMGSkpKiuLg4VatWTUFBQYqOjtYjjzyigwcP2h3NFge3bdfbbW/Qa42u1JttOmjfpt/sjoS/MDZmY3zMxdiYi7ExG+MDk5TYgmXHjh1q1qyZtmzZounTp2vbtm2aNGmSvvvuO7Vq1Up//PGH3RGL3RcPPaEr4vpq0Prluurxh/X5g4/YHQl/YWzMxviYi7ExF2NjNsan+FmWJcth0GZQe1qJLVgGDhyooKAgzZs3T9dcc42ioqJ0ww03aP78+dq9e7eGDh1qd8Ri9ee+/dq7Zp0a39lDktTglq5KS96ptOSdNicDY2M2xsdcjI25GBuzMT4wTYksWP744w/NnTtXAwYMUOnSpX2ORUREqFevXpoxY4Y8Ho9NCYufe9dula8aoVIBueswWJalkOqXKCNll83JwNiYjfExF2NjLsbGbIwPTFMiVwnbunWrPB6P6tevX+Dx+vXrKy0tTfv371dYWJjPsaysLGVlZXkfu93u85q1OOWb+itBBZvpGBuzMT7mYmzMxdiYjfGxgeXI3UxhUBZzkhgkb2aloN69+Ph4hYSEeLfIyMjijndeuKpfIvfuPTqRnS0p9zvI2L1HIZHVbU4GxsZsjI+5GBtzMTZmY3xgmhJZsNSuXVuWZWnjxo0FHt+8ebMqVqyoKlWq5Ds2ZMgQZWRkeLeUlJTzHbdYlAsLVcRljbRu+keSpI2zvlCFqEhVjI6yORkYG7MxPuZibMzF2JiN8YFpLE9JulDjbzp27KgNGzZo69atPtexpKamqlatWurbt68mTpx42vO43W6FhIQo4/dEuVzlz2fk8+7Alq2afd/DOvLHH3K6yuuWt8YrrEE9u2NBjI3pGB9zMTbmYmzMdrGNj9t9SCHhNZSRkSGXy2V3HB95P0umTR4oVxmn3XG83EeyVLH/eCO+sxJbsGzdulWtW7dW/fr1NWLECNWoUUMbNmzQU089paysLC1btkyVKlU67XkupoIFAADgYkTBUnQmFSwlsiVMkmJjY7Vy5UrVqlVLPXv2VK1atXT//ffr2muv1dKlSwtVrAAAAAA4v0rkKmF5oqOjNWXKFLtjAAAAoKSzrNzNFAZlKbEzLAAAAADMR8ECAAAAwFgluiUMAAAAMILDkbuZwqAs5iQBAAAAgJNQsAAAAAAwFi1hAAAAgN1YJcwvZlgAAAAAGIuCBQAAAICxaAkDAAAA7EZLmF/MsAAAAAAwFgULAAAAAGPREgYAAADYjRtH+mVOEgAAAAA4CQULAAAAAGPREgYAAADYjVXC/GKGBQAAAICxKFgAAAAAGIuWMAAAAMBuDit3M4VBWZhhAQAAAGAsChYAAAAAxqIlDAAAALCb5cjdTGFQFnOSAAAAAMBJKFgAAAAAGIuWMAAAAMBurBLmFzMsAAAAAIxFwQIAAADAWLSEAQAAAHazrNzNFAZlYYYFAAAAwFm7/vrr1bhxYzVp0kT/+Mc/tGbNGknSvn371KlTJ8XGxqphw4ZasmRJkc7LDAsAAACAszZz5kxVqFBBkjR79mz1799fq1at0jPPPKOWLVtqzpw5WrFihbp3767t27crIKBwpQgFCwAAAGA3y5IcBjU/nUFLWF6xIkkZGRly/PV5Zs6cqcTERElS8+bNFR4eriVLlqht27aFOi8FCwAAAIACud1un8dOp1NOp9Pv8/v27asFCxZIkubMmaODBw8qJydHoaGh3ufExMRo586dhc5gUBkHAAAAwCSRkZEKCQnxbvHx8ad8/rvvvquUlBSNGDFCTz31lCTJOmm2xuPxFCkDMywAAACA3QxdJSwlJUUul8u7+1SzK393991368EHH/Q+3r9/v3eWJTk5WVFRUYWOwgwLAAAAgAK5XC6fzV/B4na7tWfPHu/jWbNmqXLlyqpUqZJ69Oih8ePHS5JWrFih1NRUtWnTptAZmGEBAAAAcFYyMjJ022236ejRo3I4HAoNDdWXX34py7I0atQo9enTR7GxsQoKCtK0adMKvUKYRMECAAAA2M/QlrDCioyM1PLlyws8Fh4ernnz5p1xFFrCAAAAABiLggUAAACAsWgJAwAAAOxmOXI3UxiUxZwkAAAAAHASChYAAAAAxqIlDAAAALCb9ddmCoOyMMMCAAAAwFgULAAAAACMRUsYAAAAYLcL/MaR5xMFCwDbeE5k2R0BflilnHZHgB+eQzvtjoBTKVvN7gQogCcn2+4IOAu0hAEAAAAwFjMsAAAAgN1oCfOLGRYAAAAAxqJgAQAAAGAsWsIAAAAAu9ES5hczLAAAAACMRcECAAAAwFi0hAEAAAC2s/7aTGFOFmZYAAAAABiLggUAAACAsWgJAwAAAOxGR5hfzLAAAAAAMBYFCwAAAABj0RIGAAAA2I0bR/rFDAsAAAAAY1GwAAAAADAWLWEAAACA3WgJ84sZFgAAAADGomABAAAAYCxawgAAAAC70RLmFzMsAAAAAIxFwQIAAADAWLSEAQAAALaz/tpMYU4WZlgAAAAAGIuCBQAAAICxaAkDAAAA7EZHmF/MsAAAAAAwFgULAAAAAGPREgYAAADYjRtH+sUMCwAAAABjUbAAAAAAMBYtYQAAAIDdaAnzixkWAAAAAMaiYAEAAABgLFrCAAAAANtx50h/mGEBAAAAYCwKFgAAAADGoiUMAAAAsBurhPnFDAsAAAAAY1GwAAAAADAWLWEAAACA3SwZ1YZl0CJhzLAAAAAAMBcFCwAAAABj0RIGAAAAmMCgNiyTMMMCAAAAwFgULAAAAACMRUsYvA5u265Z9z6kIwcPKjgkRN3eGqew+nXtjgUxNibLzszUx/0GaP/mLQosXVrlwsPU5b8vqUJ0pN3RIP7smGzb9z/p+xfHyZPj0YnsbLUe0FdN7uhqdyxI+ubJofrtq7nK2LlL/1y+QGGX1rc7UsnAjSP9uqBmWFJSUhQXF6dq1aopKChI0dHReuSRR3Tw4EHvc9q2bSvLsmRZloKCglSrVi0NGTJEWVlZ+c63YMECdenSRaGhoQoODlatWrXUs2dP/fDDD8X5sYzxxUNP6Iq4vhq0frmuevxhff7gI3ZHwl8YG7NdcU8vPbR6sR5c+q3qdGqvLx4ebHck/IU/O2byeDz69J9DdfN/h+uB76frrvf+qy+fGqmsPw/bHQ2SGnTrov7zP1dIVHW7owCSLqCCZceOHWrWrJm2bNmi6dOna9u2bZo0aZK+++47tWrVSn/88Yf3uffdd5/27t2rbdu2afTo0Ro/fryGDx/uc74JEyaoXbt2qly5smbMmKFNmzZp2rRpat26tR577LFi/nT2+3Pffu1ds06N7+whSWpwS1elJe9UWvJOm5OBsTFbQHCwYju2k/XXb6KqX3m50pIYGxPwZ8d8me5DkqSsQ3+qTMUQBQQF2ZwIkhTdppVcl1SzOwbgdcG0hA0cOFBBQUGaN2+eSpcuLUmKiopS06ZNVatWLQ0dOlQTJ06UJJUpU0YRERHe53zwwQeaN2+e4uPjJUk7d+7Uo48+qkcffVSvvPKK9z1q1Kih1q1ba9CgQcX86ezn3rVb5atGqFRA7v8SlmUppPolykjZpYrRUTanK9kYmwvLzxPfUZ0b2tsdA+LPjsksy1L3t17SjHueUlCZYB1NP6SeU15WqaBAu6MBNrJk1jJh5mS5IGZY/vjjD82dO1cDBgzwFit5IiIi1KtXL82YMUMejyffa9euXasff/xRgYH/+0vwk08+0fHjxzV4cMFtG9YpevaysrLkdrt9totFvs9dwPcJezA2F4bF/3lNf2xPVLthz9gdBX/hz46ZcrKzteS/U3TH1Ff06Kqv1feTSZr98PM6mpZhdzQABrogCpatW7fK4/Gofv2CL/qqX7++0tLStH//fkm57V7lypWT0+lUkyZNtH//fj311FPe52/ZskUul8s7CyPlFjHlypXzbuvXry/wveLj4xUSEuLdIiMvjgtrXdUvkXv3Hp3IzpaU21+csXuPQiLpX7UbY3Nh+Om/k7Tp82/U69P3FFim9OlfgPOOPzvmSv11iw79vl9RLZpIki5peqnKR4QqdcMWe4MBMNIFUbCcTt7MSt5v0nr16qU1a9Zo6dKluv3229W/f3/ddtttPq85+bduHTt21Jo1a/TVV1/p8OHDOnHiRIHvNWTIEGVkZHi3lJSU8/CJil+5sFBFXNZI66Z/JEnaOOsLVYiKpG3CAIyN+Za+/oZ+/Wi2+nw+XcEVQuyOg7/wZ8dcrmrhcu/ZpwPbkiRJf+zYqT+SdqlyrWh7gwF2ylslzKTNEBfENSy1a9eWZVnauHGjunXrlu/45s2bVbFiRVWpUkWSFBISotq1a0uS3nvvPV166aV65513FBcXJ0mKjY1VRkaGUlNTvbMs5cqVU+3atRUQcOqvxOl0yul0nsNPZ46u48Zo9n0Pa/HosXK6yuuWt8bbHQl/YWzM5d69R/OG/FsVa0Rraufci7sDnE7du/BLm5NB4s+OqcqFVVaXl4fqo7jBsixLHo9HN44aIlfVMLujQdJXjz2j376cqz9/36d3u9yuoHJlNWj9MrtjoQSzPAVd+GGgjh07asOGDdq6davPdSypqamqVauW+vbtq4kTJ6pt27Zq0qSJXn31Ve9zEhISNGTIEG3fvl1lypTRzp07Vbt2bQ0cOFBjx471eZ+kpCTVqFFDq1evVpMmTU6by+12KyQkRBm/J8rlKn+uPi5QInhO5F9uHGawSl2cv5i5GHgOscqZ0cqyupaJ3O5DqlA19xfWLpfL7jg+8n6WTFvwslzlzGkpdv95VBWvfdKI7+yCaQkbN26csrKy1LFjR/3www9KSUnRnDlz1KFDB11yySUaOXKk39feddddsixLEyZMkJS7ctiYMWP03//+V3fffbcWLFigpKQkrVq1Sq+99pokqVSpUsXyuQAAAADb278Mbgm7YAqW2NhYrVy50ntzx1q1aun+++/Xtddeq6VLl6pSpUp+XxsUFKSHHnpIo0eP1p9//ilJevjhhzVv3jzt379f3bt3V2xsrDp37qzExETNmTNHjRo1Kq6PBgAAAMCPC6YlzFS0hAFnjpYwc9ESZi5awgxHS5iRLoiWsIVjzGsJa/uEEd/ZBXHRPQAAAHBR476Rfl0wLWEAAAAASh4KFgAAAADGoiUMAAAAsJthK3OZlIUZFgAAAADGomABAAAAYCxawgAAAADbsUyYP8ywAAAAADAWBQsAAAAAY9ESBgAAANiNVcL8YoYFAAAAgLEoWAAAAAAYi5YwAAAAwG60hPnFDAsAAAAAY1GwAAAAADAWLWEAAACA3WgJ84sZFgAAAADGomABAAAAYCwKFgAAAADGomABAAAAYCwKFgAAAADGYpUwAAAAwG6sEuYXMywAAAAAzkpmZqa6deumOnXqqEmTJurUqZOSkpIkSfv27VOnTp0UGxurhg0basmSJUU6NwULAAAAgLN2//3367ffftOaNWvUpUsX3X///ZKkZ555Ri1bttTWrVs1ZcoU9erVS9nZ2YU+LwULAAAAYLe8ljCTtiIIDg5W586dZf31upYtW2rHjh2SpJkzZ2rgwIGSpObNmys8PLxIsyxcwwIAAACgQG632+ex0+mU0+k87etee+01de3aVQcPHlROTo5CQ0O9x2JiYrRz585CZ2CGBQAAAECBIiMjFRIS4t3i4+NP+5oXX3xRW7du1ciRIyXJO+uSx+PxFCkDMywAAACA3QxdJSwlJUUul8u7+3SzKy+//LI+/fRTzZ8/X2XKlFGZMmUkSfv37/fOsiQnJysqKqrQUZhhAQAAAFAgl8vls52qYHnllVc0ffp0ffvtt6pQoYJ3f48ePTR+/HhJ0ooVK5Samqo2bdoUOgMzLAAAAADOyq5du/TEE0+oZs2auvbaayXlzsb8/PPPGjVqlPr06aPY2FgFBQVp2rRpCggofBlCwQIAAADgrFSvXt3vtSnh4eGaN2/eGZ+bggUAAACwnfXXZgpzsnANCwAAAABjUbAAAAAAMBYtYQAAAIDdDF3W2ATMsAAAAAAwFgULAAAAAGPREgYAAADYzXLkbqYwKIs5SQAAAADgJBQsAAAAAIxFSxgAAABgO24c6Q8zLAAAAACMRcECAAAAwFi0hAEAAAB2s2TUzRoN6gijYAFgH6uU0+4IwAXHKh9ldwScQs7KBLsjoACew5l2R8BZoCUMAAAAgLGYYQEAAADsxo0j/TInCQAAAACchIIFAAAAgLFoCQMAAABsx40j/WGGBQAAAICxKFgAAAAAGIuWMAAAAMBulmXYjSPNycIMCwAAAABjUbAAAAAAMBYtYQAAAIDtHDJrLsGcLOYkAQAAAICTULAAAAAAMBYtYQAAAIDdWCXML2ZYAAAAABiLggUAAACAsWgJAwAAAGxnWEuYzMnCDAsAAAAAY1GwAAAAADAWLWEAAACA7SyZ1IZlUhZmWAAAAAAYi4IFAAAAgLFoCQMAAADsZjlyN1MYlMWcJAAAAABwEgoWAAAAAMaiJQwAAACwm2XYjSMNysIMCwAAAABjUbAAAAAAMBYtYQAAAIDtuHGkP8ywAAAAADAWBQsAAAAAY9ESBgAAANiNG0f6ZU4SAAAAADgJBQsAAAAAY9ESBgAAANjMsixZBt2s0aQszLAAAAAAMBYFCwAAAABj0RIGAAAA2I4bR/rDDAsAAAAAY1GwAAAAADAWLWEAAACA3SzLqJs1ilXCAAAAAOD0KFgAAAAAGIuWMAAAAMB2rBLmDzMsAAAAAIxFwQIAAADAWLSEwevgtu2ade9DOnLwoIJDQtTtrXEKq1/X7lgQY2M6xsdcjI25GBtzZGcd1yfD39P+pN8VGBykcpXK68Ynb1OFqpXk8Xi0aMo8/frtapUKLKUyIWV19+sD7I58cbIso1bmMinLBTvD0q9fP1mWJcuyFBgYqJo1a+rJJ5/U4cOHlZSU5D128rZs2TJJUkJCgs/+8PBwde3aVRs2bLD5k9nni4ee0BVxfTVo/XJd9fjD+vzBR+yOhL8wNmZjfMzF2JiLsTHL5Te11MAPntYDUx5XbOv6+vI/H0uSln+8RPt2pOqf7z6pf777lG4d1tvmpCiJLtiCRZI6deqkvXv3aseOHRoxYoQmTJigJ5980nt8/vz52rt3r892xRVXeI+7XC7t3btXe/bs0VdffaXDhw/rxhtv1LFjx+z4OLb6c99+7V2zTo3v7CFJanBLV6Ul71Ra8k6bk4GxMRvjYy7GxlyMjVkCnIGKbVVf1l+/Ua9+abTS9hyUJP00faHaP3ijSgXmNuWUr+KyLSdKrgu6YHE6nYqIiFBkZKTuuusu9erVS7Nnz/Yer1y5siIiIny2wMBA73HLshQREaGqVauqWbNmeuyxx5ScnKzffvvNhk9jL/eu3SpfNUKlAnL/QrIsSyHVL1FGyi6bk4GxMRvjYy7GxlyMjdmWf7xEdVo3UNbhTB1J/1Obf1ivdx54Te888Jo2fLfG7ngXL8th3maIi+oaltKlS+v48eNn9Nr09HR98MEHkuRT1JwsKytLWVlZ3sdut/uM3s9E1sm9ih6PPUGQD2NjNsbHXIyNuRgbMy1+9zsd3HVAfV99QMezjuvE8RM6nnVccW8MUsbvaZr84OsKrRGusJpV7Y6KEuSiKViWL1+uDz74QO3atfPua926tRwO3+owIyNDpUqV8v53uXLl5PF4dOTIEUnSTTfdpHr16vl9n/j4eL3wwgvn4RPYy1X9Erl379GJ7GyVCgiQx+NRxu49Comsbne0Eo+xMRvjYy7GxlyMjZl+mr5Qm39Yrz5jH1BgcJACg4MUVNqpxh1z2+lDwisqslGM9mzeRcGCYmXOXM8Z+PLLL1WuXDkFBwerVatWuvrqq/X66697j8+YMUNr1qzx2fKKFUkqX7681qxZo19++UWTJk1SrVq1NGnSpFO+55AhQ5SRkeHdUlJSztvnK07lwkIVcVkjrZv+kSRp46wvVCEqUhWjo2xOBsbGbIyPuRgbczE25ln64SJtmL9avcfer+Dypb37G7Zvom0/b5YkHT10RLs3pSi8FsXK+WEZuJnhgp5hufbaazVx4kQFBgaqWrVq3laupKQkSVJkZKRq167t9/UOh8N7vF69ekpNTVXPnj31ww8/+H2N0+mU0+k8dx/CIF3HjdHs+x7W4tFj5XSV1y1vjbc7Ev7C2JiN8TEXY2MuxsYc7n3p+nb8F6pYrbLeHZT7i9tSgaV075uP6Lr7O+uz+A+1ctZPkqQ2va9T1brMhKF4WR7Phdk02q9fP6Wnp/tcZJ8nKSlJNWrU0OrVq9WkSZMCX5+QkKBHH31U6enp3n1ut1uRkZFKSEjQLbfcUqgcbrdbISEhyvg9US5X+TP4JAAA4GKRszLB7ggogPtwpip2+pcyMjLkcpm10lnez5LpGz6Tq3xZu+N4uQ8dVoVLbzbiO7ugW8JO5+DBg0pNTfXZMjMz/T7f5XLp3nvv1bBhw3SB1nEAAAC4EOXdONKkzRAXdcHSvn17Va1a1WcraEbm7x555BFt2rRJH330UfGEBAAAAODXBXsNS0JCgt9jMTExp50h6devn/r165dvf1RU1BkvjQwAAADg3LpgCxYAAADgomHYzRpNymJOEgAAAAA4CQULAAAAAGPREgYAAADYzqybNZqUhRkWAAAAAMaiYAEAAABgLFrCAAAAALsZdrNGk7IwwwIAAADAWBQsAAAAAIxFSxgAAABgO4fMmkswJ4s5SQAAAADgJBQsAAAAAIxFSxgAAABgN1YJ84sZFgAAAADGomABAAAAYCxawgAAAAC70RLmFzMsAAAAAIxFwQIAAADgrAwaNEgxMTGyLEu//vqrd/++ffvUqVMnxcbGqmHDhlqyZEmRz03BAgAAANjOYeBWeN27d9eSJUsUHR3ts/+ZZ55Ry5YttXXrVk2ZMkW9evVSdnZ2kc7NNSwAAAAAzsrVV19d4P6ZM2cqMTFRktS8eXOFh4dryZIlatu2baHPTcECAAAAoEBut9vnsdPplNPpLNRrDx48qJycHIWGhnr3xcTEaOfOnUXKQEsYAAAAYLe8VcJM2iRFRkYqJCTEu8XHxxfxY/muNubxeIr81TDDAgAAAKBAKSkpcrlc3seFnV2RpMqVK0uS9u/f751lSU5OVlRUVJEyMMMCAAAAoEAul8tnK0rBIkk9evTQ+PHjJUkrVqxQamqq2rRpU6RzMMMCAAAAGMGcmzUW1cCBA/XZZ58pNTVV7du3V7ly5bRt2zaNGjVKffr0UWxsrIKCgjRt2jQFBBStBKFgAQAAAHBWxo8f751J+bvw8HDNmzfvrM5NSxgAAAAAYzHDAgAAANjtbytzGcGgLMywAAAAADAWBQsAAAAAY9ESBgAAANjOklmrhJmThRkWAAAAAMaiYAEAAABgLFrCAAAAALuxSphfzLAAAAAAMBYFCwAAAABj0RIGAAAA2M4hs+YSzMliThIAAAAAOAkFCwAAAABj0RIGAAAA2I1VwvxihgUAAACAsShYAAAAABiLljAAAADAdtZfmynMycIMCwAAAABjMcNyljwejyTJfeiQzUkAAIDdcg5n2h0BBXD/NS55P7fhwkLBcpYO/VWoRNZubHMSAAAAnMqhQ4cUEhJid4yCsUqYXxQsZ6latWpKSUlR+fLlZRk0sGfK7XYrMjJSKSkpcrlcdsfB3zA25mJszMb4mIuxMdfFNjYej0eHDh1StWrV7I6CM0DBcpYcDoeqV69ud4xzzuVyXRR/QV2MGBtzMTZmY3zMxdiY62IaG2NnVnBaFCwAAACA7VglzB9WCQMAAABgLAoW+HA6nRo2bJicTqfdUXASxsZcjI3ZGB9zMTbmYmxgEsvD+m4AAACALdxut0JCQpSeuFQuVzm743i53X+qQo1WysjIsP06JmZYAAAAABiLggUAAACAsVglDAAAALAdq4T5wwwLAAAAAGNRsAAALloHDhzQwYMH7Y4BADgLFCwAgItKenq6Bg4cqCpVqig8PFxhYWGqUqWKHnroIaWnp9sdD3/ZsGGD1q1b5902bNhgdyTAXpZl3mYICpYSzu12KycnJ9/+EydOyO1225AIZ8rlcmnHjh12xygRtm7dqjvvvLPAPyMZGRm66667GAub/PHHH2rRooWmTp2q2267TWPGjNHLL7+sW2+9VQkJCWrVqpXS0tLsjlkiLV68WM2bN/c+btmypZo2baomTZqoSZMmaty4sebPn29jQpzOxx9/bHcElFAULCXYrFmz1KxZM2VmZuY7lpWVpebNm+uLL76wIRnOBLdUKj7/+c9/FBkZWeC69CEhIYqMjNR//vMfG5Lh3//+t4KCgrR9+3a98cYbevTRR/XYY4/pzTff1LZt2xQYGKh///vfdscskSZMmKA+ffr47FuwYIESExO1Y8cOPfLII5o4caJN6SBJ2dnZ2rBhg7Zs2eKz/7PPPtNll12mXr162ZQMJR0FSwk2ceJEDR48WGXKlMl3rEyZMnr66ac1btw4G5IBZvvhhx/Uo0cPv8dvv/12ff/998WYCHlmz56tl19+WeHh4fmORUREaPTo0Zo1a5YNybBixQpdeeWVPvuqV6+u6OhoxcTEqE+fPlq6dKlN6bBx40bVqVNHjRs3Vv369XXrrbfq999/1zXXXKO7775bHTp00LZt2+yOeZFzGLiZwZwkKHa//vqr2rZt6/f41VdfrfXr1xdfIOACkZycrLCwML/Hq1SpopSUlGJMhDx79+7VpZde6vd4w4YNlZqaWoyJkGf37t2qWrWq9/HUqVMVERHhfVypUiUWSLDRM888oxo1auizzz7T7bffrtmzZ+sf//iH2rVrp5SUFL388suKjIy0OyZKKAqWEiwtLU3Z2dl+jx8/fpxeb6AAISEh2r59u9/j27ZtK7BdDOdflSpVlJSU5Pd4YmKiKleuXHyB4FW+fHklJiZ6H996660+M/yJiYn8ubHR8uXL9Z///EddunTxtuY99dRTev7551W+fHmb06Gko2ApwWJiYrRy5Uq/x1euXKno6OhiTARcGK6++mq9/vrrfo+/9tpr+sc//lGMiZCnU6dOGjp0qI4dO5bvWFZWlp577jl16tTJhmRo0aKF3n33Xb/HExIS1KJFi2JMhL/bt2+fLrnkEklShQoVVKZMGV1zzTU2pyph7F4RzOBVwrjTfQl26623aujQoerQoUO+fu/U1FT961//Uu/evW1KB5hryJAhatWqlbp3767Bgwerbt26kqTNmzdr9OjRmjt3rn766SebU5ZML7zwgpo1a6bY2FgNHDhQ9erVk5Tbnz9hwgRlZWVp2rRpNqcsmR5//HG1b99elStX1lNPPeVtq9y3b59GjRql9957T/PmzbM5ZcllWZYcjv/9HtvhcCgwMNDGRMD/WB6WFiqxDh06pFatWmnnzp3q3bu36tatK8uytGnTJr3//vuKjIzUsmXLmAq+QAQEBGjt2rWn7N/HufPll1+qf//++XruK1eurLfffls33XSTTcmQmJioAQMGaN68ed7V8yzLUocOHTRu3DjVrl3b5oQl14QJE/TYY48pOztbLpdLlmUpIyNDAQEBGjNmjB566CG7I5ZYDodDISEhsv76rXp6erpcLpdPESPlLh2Oc8vtdiskJETpySvlcpWzO46X2/2nKkQ3U0ZGhu3tmhQsJVxGRoaGDBmiGTNmeK9XqVixonr27KkXX3xRFSpUsDcgCq18+fJau3atatasaXeUEuPo0aOaM2eOtm3bJo/Hozp16uj6668vcOU9FL+0tDRt3bpVklS7dm1VqlTJ5kSQpJSUFH388cfesYmNjVX37t25oNtmU6dOLdTz7r777vOcpOT5X8Hyi4EFyxUULDCHx+PRgQMH5PF4FBoa6v0NCy4cFCwALnRHjhyh4EeJQ8Fyelx0D0nS+vXrtWjRIi1evFi//vqr3XEAo/3888/65ptvfPa9++67qlGjhsLCwnT//fcrKyvLpnTAhSczM1NjxozhFy42Wr58uU6cOOF9fPLvs7OysjRz5szijgVIomAp8ZYvX65GjRqpadOmuv3229WjRw81adJEjRs31ooVK+yOBxhp+PDhWrdunffx+vXrFRcXp/bt2+uZZ57RF198ofj4eBsTAuY5duyYhg4dqubNm6t169aaPXu2JGnKlCmqWbOmXnnlFT3yyCP2hizBWrVq5XNNXkhIiHbs2OF9nJ6erjvvvNOOaCWIZeBmBgqWEmzjxo1q166dSpcurffee0+rVq3SL7/8omnTpsnpdKpdu3bauHGj3TFRSLTxFZ81a9aoXbt23scffvihWrRoobfeekuPP/64XnvtNX4TCZxk+PDhGjdunKKjo5WYmKgePXrogQce0EsvvaT4+HglJSVpyJAhdscssU6eUSnoigGuIoBdWNa4BBs2bJg6dOigTz75xOeH3aZNm+rOO+/UrbfequHDh/OD1wWCf0iKT1pams9S4IsWLfK5t0fz5s250z1wkpkzZyohIUG33HKL1q5dq6ZNm8rtdmvDhg0KCODHkQsBvxiDXZhhKcEWLlyoZ599tsC/gCzL0rPPPqsFCxbYkAx53G63cnJy8u0/ceKE3G63z75vvvnGe9MvnF/h4eHeO3YfO3ZMq1atUqtWrbzHDx06xP0LgJOkpKSoefPmkqTLLrtMQUFBevrppylWgL/k3qvRMmiz+xv5HwqWEuzQoUP5bhj5dxERETp06FAxJsLfzZo1S82aNVNmZma+Y1lZWWrevLm++OIL7742bdrI6XQWZ8QSq1OnTnrmmWe0ePFiDRkyRGXKlPG5s/26detUq1YtGxMC5jl+/LiCgoK8jwMDAxUSEmJjIpxs48aNWrdundatWyePx6PNmzd7H2/YsMHueCjB+LVGCRYTE6Ply5f7Xfv+559/VnR0dDGnQp6JEydq8ODBBS7xWaZMGT399NMaN26cunbtakO6km3EiBG69dZbdc0116hcuXKaOnWqzw9ikydP1vXXX29jQsBMzz//vPfvtGPHjmnEiBH5ipZXXnnFjmiQdN111/k87tKli6Tc3/p7PB5awmAbCpYSrGfPnnr88cdVt25dNWzY0OfY+vXr9eSTT3KDKBv9+uuvmjBhgt/jV199tf71r38VYyLkCQ0N1eLFi5WRkaFy5cqpVKlSPsc/+ugjlStnzlr6gAmuvvpq/fbbb97HrVu39lmFSuIaCTvltbnCTmatzGVSFgqWEmzIkCGaP3++mjRpog4dOqh+/fqScqeE58+fryuvvJIVW2yUlpam7Oxsv8ePHz+utLS0YkyEk/lrZ+GO6kB+Cxcu9Hl84MABWZalypUr2xMIPsLCwvTkk09q9uzZOn78uNq3b6/XXntNVapUsTsaQMFSkgUHB2vBggUaO3aspk+frkWLFkmS6tSpoxEjRqhXr14aMGCAJk+ebHPSkikmJkYrV65UvXr1Cjy+cuVKWvZscuuttxbqeZ9++ul5TgJcWNLT0zV06FDNmDHD+wuXihUr6o477tCIESNUoUIFewOWYM8//7wSEhLUq1cvBQcHa/r06frnP/+pjz76yO5ogCwPa6HCj7Vr1+ryyy/3ufMtis/QoUP13nvvafny5fkWR0hNTVWLFi3Uu3dvjRw50qaEJdc999xTqOdNmTLlPCcBLhx//PGHWrVqpd27d6tXr16qX7++PB6PNm3apA8++ECRkZH66aefVLFiRbujlki1atXSyJEjdccdd0jKvbH0VVddpczMzHxtrzi33G63QkJClJGyRi5XebvjeLndhxQS2UQZGRlyuVy2ZqFggV8ULPY6dOiQWrVqpZ07d6p3796qW7euLMvSpk2b9P777ysyMlLLli1T+fLm/OUGAP48+uij+u677zR//vwCfwlz/fXXq127dho7dqxNCUu2oKAgJSYm+iyPX7p0aW3ZssXv4jw4NyhYTo+WMMBQ5cuX148//qghQ4bka5/o3bu3XnzxRYoVm/Tv3/+0z7EsS++8804xpAEuDLNnz9Ybb7xR4HL6ERERGj16tB588EEKFpucOHHCZ7VDSQoICDjltZRAcaFgAQwWEhKiCRMmaPz48Tpw4IA8Ho9CQ0NZScdmCQkJio6OVtOmTcUkNVA4e/fu1aWXXur3eMOGDZWamlqMifB3Ho9H/fr187mfV2Zmph588EGVLVvWu49r884nVgnzh4KlBDvdhcPp6enFEwSntX79em3ZskWWZalOnTpq1KiR3ZFKtAcffFAffvihduzYof79+6t3796sDAacRpUqVZSUlKTq1asXeDwxMZEVw2xU0G0MevfubUMSID+uYSnBuHDYfMuXL1dcXJw2btzo/U2+ZVm69NJL9c4776h58+Y2Jyy5srKy9Omnn2ry5Mn66aefdOONNyouLk7XX389M2BAAeLi4rRt2zZ9++23+VqPsrKy1LFjR9WqVYtWSpQ4/7uGZa2B17BcZsQ1LBQsgKE2btyoFi1aqH79+nrsscd8VtQZO3asfvvtNy1btkwNGjSwO2qJl5ycrISEBL377rs6fvy4Nm7cyI0jgZPs2rVLzZo1k9Pp1MCBA71Ltm/cuFETJkxQVlaWVq5cyQXeKHG8Bcuu9eYVLNUbUbAA8K9Hjx46ceKEPvnkk3y/sfd4PLr11lsVGBiomTNn2pQQeXbu3KmEhAQlJCTo2LFj2rx5MwULUIDExEQNGDBA8+bN85k17tChg8aNG6fatWvbnBAofhQsp0fBAhgqNDRU33zzjZo1a1bg8RUrVqhz587av39/MSeD5NsStmTJEnXp0kX33HOPOnXqJIfDYXc8wGhpaWnaunWrJKl27dpcA4YSjYLl9LjoHjDUoUOHClz+M09ERIQOHTpUjImQZ8CAAfrwww8VFRWle+65Rx9++CEXCwNFULFiRV155ZV2xwAMwyph/lCwAIaKiYnR8uXL/fZz//zzz4qOji7mVJCkSZMmKSoqSjVq1NCiRYu0aNGiAp/H8p8AAJw9ChbAUD179tTjjz+uunXrqmHDhj7H1q9fryeffLLAZShx/vXt25eVwAAAKCZcwwIYKjMzU+3atdPPP/+sDh06qH79+pJyV9SZP3++rrzySn3//fcKDg62OSkAADhT3mtYdm8w7xqWSy414hoWrgwFDBUcHKwFCxZo5MiR2rt3ryZNmqRJkyYpNTVVI0aM0IwZMzRgwAC7YwIAAJxXzLAAF6i1a9fq8ssv14kTJ+yOAgAAzhAzLKfHNSwAAACA7VglzB9awgAAAAAYi4IFAAAAgLFoCQMMdeutt57yeHp6evEEAQAA559l5W6mMCgLBQtgqJCQkNMe79u3bzGlAQAAsAcFC2CoKVOm2B0BAADAdhQsAAAAgO1YJcwfLroHAAAAYCwKFgAAAADGomABgAvQ8OHD1aRJE+/jfv36qVu3bsWeIykpSZZlac2aNX6fExMTo1dffbXQ50xISFCFChXOOptlWZo9e/ZZnwcAikXeKmEmbYagYAGAc6Rfv36yLEuWZSkwMFA1a9bUk08+qcOHD5/39/7vf/+rhISEQj23MEUGAACm4KJ7ADiHOnXqpClTpuj48eNavHix7r33Xh0+fFgTJ07M99zjx48rMDDwnLzv6ZbBBgDgQsUMCwCcQ06nUxEREYqMjNRdd92lXr16eduS8tq4Jk+erJo1a8rpdMrj8SgjI0P333+/wsLC5HK5dN1112nt2rU+533ppZcUHh6u8uXLKy4uTpmZmT7HT24Jy8nJ0ahRo1S7dm05nU5FRUVp5MiRkqQaNWpIkpo2bSrLstS2bVvv66ZMmaL69esrODhY9erV04QJE3zeZ/ny5WratKmCg4PVrFkzrV69usjf0SuvvKJGjRqpbNmyioyM1IABA/Tnn3/me97s2bNVp04dBQcHq0OHDkpJSfE5/sUXX+iKK65QcHCwatasqRdeeEHZ2dlFzgMAZrAM3MxAwQIA51Hp0qV1/Phx7+Nt27Zp5syZ+uSTT7wtWTfeeKNSU1P19ddf65dfftHll1+udu3a6Y8//pAkzZw5U8OGDdPIkSO1cuVKVa1aNV8hcbIhQ4Zo1KhReu6557Rx40Z98MEHCg8Pl5RbdEjS/PnztXfvXn366aeSpLfeektDhw7VyJEjtWnTJr344ot67rnnNHXqVEnS4cOH1aVLF9WtW1e//PKLhg8frieffLLI34nD4dBrr72mX3/9VVOnTtX333+vwYMH+zznyJEjGjlypKZOnaoff/xRbrdbd9xxh/f43Llz1bt3bw0aNEgbN27UG2+8oYSEBG9RBgC4iHgAAOfE3Xff7bn55pu9j3/++WdP5cqVPbfffrvH4/F4hg0b5gkMDPTs27fP+5zvvvvO43K5PJmZmT7nqlWrlueNN97weDweT6tWrTwPPvigz/EWLVp4LrvssgLf2+12e5xOp+ett94qMGdiYqJHkmf16tU++yMjIz0ffPCBz77/+7//87Rq1crj8Xg8b7zxhqdSpUqew4cPe49PnDixwHP9XXR0tGfs2LF+j8+cOdNTuXJl7+MpU6Z4JHmWLVvm3bdp0yaPJM/PP//s8Xg8nn/84x+eF1980ec806ZN81StWtX7WJJn1qxZft8XAEyQkZHhkeTJ2LvF4zm815gtY++W3FwZGXZ/RR6uYQGAc+jLL79UuXLllJ2drePHj+vmm2/W66+/7j0eHR2t0NBQ7+NffvlFf/75pypXruxznqNHj2r79u2SpE2bNunBBx/0Od6qVSstWLCgwAybNm1SVlaW2rVrV+jc+/fvV0pKiuLi4nTfffd592dnZ3uvj9m0aZMuu+wylSlTxidHUS1YsEAvvviiNm7cKLfbrezsbGVmZurw4cMqW7asJCkgIEDNmjXzvqZevXqqUKGCNm3apCuvvFK//PKLVqxY4TOjcuLECWVmZurIkSM+GQHggmA5cjdTGJSFggUAzqFrr71WEydOVGBgoKpVq5bvovq8H8jz5OTkqGrVqlq4cGG+c53p0r6lS5cu8mtycnIk5baFtWjRwudYqVKlJEkej+eM8vxdcnKyOnfurAcffFD/93//p0qVKmnJkiWKi4vzaZ2TcpclPlnevpycHL3wwgu69dZb8z0nODj4rHMCAMxBwQIA51DZsmVVu3btQj//8ssvV2pqqgICAhQTE1Pgc+rXr69ly5apb9++3n3Lli3ze87Y2FiVLl1a3333ne699958x4OCgiTlzkjkCQ8P1yWXXKIdO3aoV69eBZ63QYMGmjZtmo4ePeotik6VoyArV65Udna2xowZI4cj97d3M2fOzPe87OxsrVy5UldeeaUk6bffflN6errq1asnKfd7++2334r0XQMALkwULABgo/bt26tVq1bq1q2bRo0apbp162rPnj36+uuv1a1bNzVr1kyPPPKI7r77bjVr1kxt2rTR+++/rw0bNqhmzZoFnjM4OFhPP/20Bg8erKCgIF111VXav3+/NmzYoLi4OIWFhal06dKaM2eOqlevruDgYIWEhGj48OEaNGiQXC6XbrjhBmVlZWnlypVKS0vT448/rrvuuktDhw5VXFyc/vWvfykpKUkvv/xykT5vrVq1lJ2drddff11du3bVjz/+qEmTJuV7XmBgoB5++GG99tprCgwM1EMPPaSWLVt6C5jnn39eXbp0UWRkpHr06CGHw6F169Zp/fr1GjFiRNEHAgBsZ9bKXCZlMac5DQBKIMuy9PXXX+vqq69W//79VadOHd1xxx1KSkryrurVs2dPPf/883r66ad1xRVXKDk5Wf/85z9Ped7nnntOTzzxhJ5//nnVr19fPXv21L59+yTlXh/y2muv6Y033lC1atV08803S5Luvfdevf3220pISFCjRo10zTXXKCEhwbsMcrly5fTFF19o48aNatq0qYYOHapRo0YV6fM2adJEr7zyikaNGqWGDRvq/fffV3x8fL7nlSlTRk8//bTuuusutWrVSqVLl9aHH37oPd6xY0d9+eWX+vbbb9W8eXO1bNlSr7zyiqKjo4uUBwBgPstzLpqSAQAAABSZ2+1WSEiIMlK3y+Uqb3ccL7f7kEIiaikjI0Mul8vWLLSEAQAAAHazrNzNFAZloSUMAAAAgLEoWAAAAACcta1bt6p169aqU6eOrrzySm3cuPGcnJeCBQAAALCdZeBWNA888IDuv/9+bdmyRYMHD1ZcXFyRz1EQChYAAAAAZ2Xfvn1atWqVevfuLUm67bbblJiYqKSkpLM+NxfdAwAAADZzuw/ZHcFHXh632+2z3+l0yul05nt+SkqKqlWrpoCA3PLCsixFRUVp586dfm+MXFgULAAAAIBNgoKCFBERocjYxnZHyadcuXKKjIz02Tds2DANHz68wOdbJ60sdq7unkLBAgAAANgkODhYiYmJOnbsmN1R8vF4PPmKkIJmVyQpMjJSu3btUnZ2tgICAuTxeJSSkqKoqKizzkHBAgAAANgoODhYwcHBdsc4K2FhYWratKnee+899evXT5988oliYmLOuh1M4k73AAAAAM6B3377Tf369dPBgwflcrk0depUXXrppWd9XgoWAAAAAMZiWWMAAAAAxqJgAQAAAGAsChYAAAAAxqJgAQAAAGAsChYAAAAAxqJgAQAAAGAsChYAAAAAxqJgAQAAAGAsChYAAAAAxqJgAQAAAGAsChYAAAAAxvp/AeccCq/ln/8AAAAASUVORK5CYII=",
+ "text/plain": [
+ "<Figure size 1000x1000 with 2 Axes>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plot_confusion_matrix(cm, \n",
+ " cm_labels, \n",
+ " show=True, \n",
+ " save=False, \n",
+ " img_path=None, \n",
+ " normalize=None, \n",
+ " decimal_places=2, \n",
+ " figsize=(10,10), \n",
+ " SMALL_SIZE=8, \n",
+ " MEDIUM_SIZE=12, \n",
+ " BIGGER_SIZE=14, \n",
+ " cmap='OrRd', \n",
+ " xticks_rotation='vertical', \n",
+ " title='Confusion Matrix')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "26afeb9c-bfba-4733-b894-611db84e5347",
+ "metadata": {},
+ "source": [
+ "## Evaluation mit nervaluate\n",
+ "\n",
+ "Die Python-Bibliothek [nervaluate](https://github.com/MantisAI/nervaluate) bietet eine noch detailliertere Evaluation der NER-Ergebnisse. Die Grundlage für das Paket ist dieser [Blogpost](https://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/) von David Batista. Hier werden auch die Auswertungsoptionen gut erläutert.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 55,
+ "id": "c6f08bb3-92e3-4bba-8b22-08bee2e64496",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "tags = [ 'PER', 'ORG', 'LOC', 'MISC' ]\n",
+ "evaluator = Evaluator(iob_true, \n",
+ " spacy_iob_pred,\n",
+ " tags=tags, \n",
+ " loader='list')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "id": "78e708de-b28d-4713-b75e-0c9ea21df915",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Evaluation\n",
+ "results, results_by_tag = evaluator.evaluate()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "id": "5d65f4b7-0237-41e9-a95e-9e41ee9726f7",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'ent_type': {'actual': 117,\n",
+ " 'correct': 113,\n",
+ " 'f1': 0.9535864978902954,\n",
+ " 'incorrect': 4,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 120,\n",
+ " 'precision': 0.9658119658119658,\n",
+ " 'recall': 0.9416666666666667,\n",
+ " 'spurious': 0},\n",
+ " 'exact': {'actual': 117,\n",
+ " 'correct': 115,\n",
+ " 'f1': 0.970464135021097,\n",
+ " 'incorrect': 2,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 120,\n",
+ " 'precision': 0.9829059829059829,\n",
+ " 'recall': 0.9583333333333334,\n",
+ " 'spurious': 0},\n",
+ " 'partial': {'actual': 117,\n",
+ " 'correct': 115,\n",
+ " 'f1': 0.9789029535864978,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 3,\n",
+ " 'partial': 2,\n",
+ " 'possible': 120,\n",
+ " 'precision': 0.9914529914529915,\n",
+ " 'recall': 0.9666666666666667,\n",
+ " 'spurious': 0},\n",
+ " 'strict': {'actual': 117,\n",
+ " 'correct': 113,\n",
+ " 'f1': 0.9535864978902954,\n",
+ " 'incorrect': 4,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 120,\n",
+ " 'precision': 0.9658119658119658,\n",
+ " 'recall': 0.9416666666666667,\n",
+ " 'spurious': 0}}\n"
+ ]
+ }
+ ],
+ "source": [
+ "pprint(results)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 58,
+ "id": "7caef7c0-040a-4141-aa48-f9d44fdcfd92",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'LOC': {'ent_type': {'actual': 80,\n",
+ " 'correct': 79,\n",
+ " 'f1': 0.9693251533742331,\n",
+ " 'incorrect': 1,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 83,\n",
+ " 'precision': 0.9875,\n",
+ " 'recall': 0.9518072289156626,\n",
+ " 'spurious': 0},\n",
+ " 'exact': {'actual': 80,\n",
+ " 'correct': 79,\n",
+ " 'f1': 0.9693251533742331,\n",
+ " 'incorrect': 1,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 83,\n",
+ " 'precision': 0.9875,\n",
+ " 'recall': 0.9518072289156626,\n",
+ " 'spurious': 0},\n",
+ " 'partial': {'actual': 80,\n",
+ " 'correct': 79,\n",
+ " 'f1': 0.9754601226993865,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 3,\n",
+ " 'partial': 1,\n",
+ " 'possible': 83,\n",
+ " 'precision': 0.99375,\n",
+ " 'recall': 0.9578313253012049,\n",
+ " 'spurious': 0},\n",
+ " 'strict': {'actual': 80,\n",
+ " 'correct': 79,\n",
+ " 'f1': 0.9693251533742331,\n",
+ " 'incorrect': 1,\n",
+ " 'missed': 3,\n",
+ " 'partial': 0,\n",
+ " 'possible': 83,\n",
+ " 'precision': 0.9875,\n",
+ " 'recall': 0.9518072289156626,\n",
+ " 'spurious': 0}},\n",
+ " 'MISC': {'ent_type': {'actual': 0,\n",
+ " 'correct': 0,\n",
+ " 'f1': 0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 0,\n",
+ " 'precision': 0,\n",
+ " 'recall': 0,\n",
+ " 'spurious': 0},\n",
+ " 'exact': {'actual': 0,\n",
+ " 'correct': 0,\n",
+ " 'f1': 0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 0,\n",
+ " 'precision': 0,\n",
+ " 'recall': 0,\n",
+ " 'spurious': 0},\n",
+ " 'partial': {'actual': 0,\n",
+ " 'correct': 0,\n",
+ " 'f1': 0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 0,\n",
+ " 'precision': 0,\n",
+ " 'recall': 0,\n",
+ " 'spurious': 0},\n",
+ " 'strict': {'actual': 0,\n",
+ " 'correct': 0,\n",
+ " 'f1': 0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 0,\n",
+ " 'precision': 0,\n",
+ " 'recall': 0,\n",
+ " 'spurious': 0}},\n",
+ " 'ORG': {'ent_type': {'actual': 11,\n",
+ " 'correct': 8,\n",
+ " 'f1': 0.7272727272727273,\n",
+ " 'incorrect': 3,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 11,\n",
+ " 'precision': 0.7272727272727273,\n",
+ " 'recall': 0.7272727272727273,\n",
+ " 'spurious': 0},\n",
+ " 'exact': {'actual': 11,\n",
+ " 'correct': 10,\n",
+ " 'f1': 0.9090909090909091,\n",
+ " 'incorrect': 1,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 11,\n",
+ " 'precision': 0.9090909090909091,\n",
+ " 'recall': 0.9090909090909091,\n",
+ " 'spurious': 0},\n",
+ " 'partial': {'actual': 11,\n",
+ " 'correct': 10,\n",
+ " 'f1': 0.9545454545454546,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 1,\n",
+ " 'possible': 11,\n",
+ " 'precision': 0.9545454545454546,\n",
+ " 'recall': 0.9545454545454546,\n",
+ " 'spurious': 0},\n",
+ " 'strict': {'actual': 11,\n",
+ " 'correct': 8,\n",
+ " 'f1': 0.7272727272727273,\n",
+ " 'incorrect': 3,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 11,\n",
+ " 'precision': 0.7272727272727273,\n",
+ " 'recall': 0.7272727272727273,\n",
+ " 'spurious': 0}},\n",
+ " 'PER': {'ent_type': {'actual': 26,\n",
+ " 'correct': 26,\n",
+ " 'f1': 1.0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 26,\n",
+ " 'precision': 1.0,\n",
+ " 'recall': 1.0,\n",
+ " 'spurious': 0},\n",
+ " 'exact': {'actual': 26,\n",
+ " 'correct': 26,\n",
+ " 'f1': 1.0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 26,\n",
+ " 'precision': 1.0,\n",
+ " 'recall': 1.0,\n",
+ " 'spurious': 0},\n",
+ " 'partial': {'actual': 26,\n",
+ " 'correct': 26,\n",
+ " 'f1': 1.0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 26,\n",
+ " 'precision': 1.0,\n",
+ " 'recall': 1.0,\n",
+ " 'spurious': 0},\n",
+ " 'strict': {'actual': 26,\n",
+ " 'correct': 26,\n",
+ " 'f1': 1.0,\n",
+ " 'incorrect': 0,\n",
+ " 'missed': 0,\n",
+ " 'partial': 0,\n",
+ " 'possible': 26,\n",
+ " 'precision': 1.0,\n",
+ " 'recall': 1.0,\n",
+ " 'spurious': 0}}}\n"
+ ]
+ }
+ ],
+ "source": [
+ "pprint(results_by_tag)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}