Once officially published, this 1 million subcorpus of the National Corpus of Polish will be available publicly for free. Until a specific license is proposed, the distribution of the corpus of parts thereof is prohibited.
The origin of texts in NKJP may be:
See sourceDesc/bibl/note[@text_origin] in particular header.xml files.
A linguistic corpus is a collection of texts where one can find the typical use of a single word or a phrase, as well as their meaning and grammatical function. Nowadays, without access to a language corpus, it has become impossible to do linguistic research, to write dictionaries, grammars and language teaching books, to create search engines sensitive to Polish inflexion, machine translation engines and software of advanced language technology. Language corpora have become an essential tool for linguists, but they are also helpful for software engineers, scholars of literature and culture, historians, librarians and other specialists of art and computer sciences.
There already exist national corpora compiled by the British, Germans, Czech and Russians. Polish people also need an extensive, well balanced language corpus – a language source which can be accessed online.
The National Corpus of Polish is a shared initiative of four institutions: Institute of Computer Science at the Polish Academy of Sciences (coordinator), Institute of Polish Language at the Polish Academy of Sciences, Polish Scientific Publishers PWN, and the Department of Computational and Corpus Linguistics at the University of Łódź. It has been registered as a research-development project of the Ministry of Science and Higher Education.
These four institutions have started cooperation to build a reference corpus of Polish language containing hundreds millions of words. The corpus that will appear soon on this site will be searchable by means of advanced tools that analyse Polish inflection and the Polish sentence structure.
The list of sources for the corpora contains classic literature, daily newspapers, specialist periodicals and journals, transcripts of conversations, and a variety of short-lived and internet texts. For a corpus to be reliable, not only it is necessary to contain a high number of words, but it also needs a diversity of texts with respect to the subject and genre. The conversations ought to represent both male and female speakers, in various age groups, coming from various regions in Poland.
For each text type, words in all texts of this type are counted and percent of each text to be sampled is determined (so that the target subcorpus has assumed proportions). Newspaper articles etc. are grouped in aggregates (sample of an article would be too short to be sensible).
For each text (or aggregate), each paragraph's length in words is counted and sensible paragraphs are marked. For paragraphs shorter than 25 words the sensibility condition is "begins like a sentence and ends like a sentence". For longer paragraphs, it is just "begins like a sentence". The exact regular expressions to be matched are
/^\s*[\(\"]?(-\s|\s)?([0-9A-Z]|Ą|Ć|Ę|Ł|Ó|Ń|Ś|Ź|Ż).*[\.!?\x{2026}\x{2025}:]+[\"\)]*\s*$
and
/^\s*[\(\"]?(-\s|\s)?([0-9A-Z]|Ą|Ć|Ę|Ł|Ó|Ń|Ś|Ź|Ż).*$.
Then comes the loop:
* Randomly select a paragraph
* Look at all sensible sequences of paragraphs, 40 to 70 words in length, containing this paragraph. Sensible sequence contains only sensible, unused paragraphs, has average paragraph length greater than 5, and all paragraphs come from the same source (important in aggregates: we don't step on source file boundaries).
* Choose the sequence with the length closest to 55 words and add it to the sample.
* Mark all paragraph in the chosen sequence as used.
...until we have enough words in all the samples from the given file.
The algorithm is deliberately simple and does not attempt to anticipate and deal with all possible problems. Inevitably, in many real-life cases it requires a manual intervention anyway. Some manually altered samples may be shorter than 40 or longer than 70 words, but in only a few cases fall out of the 25-100 words interval.
For privacy reasons, some of the people's names mentioned in the transcribed conversations have been modified. Uppercase characters are only used in popular proper nouns.