For more information see TREC CAR website.
| Rprec/manual | map/manual | ndcg/manual | |
|---|---|---|---|
| uog-heading-rh-sdm | 0.349 ± 0.013 | 0.354 ± 0.013 | 0.577 ± 0.014 |
| UNH-p-l2r | 0.335 ± 0.013 | 0.342 ± 0.014 | 0.577 ± 0.014 |
| uog-linear-ltr-hier | 0.324 ± 0.012 | 0.328 ± 0.013 | 0.564 ± 0.014 |
| UNH-p-mixed | 0.318 ± 0.012 | 0.294 ± 0.012 | 0.469 ± 0.014 |
| UNH-p-sdm | 0.313 ± 0.013 | 0.295 ± 0.013 | 0.472 ± 0.014 |
| guir-exp | 0.285 ± 0.012 | 0.292 ± 0.012 | 0.510 ± 0.013 |
| entityEmbedLambdaMart | 0.280 ± 0.011 | 0.279 ± 0.012 | 0.522 ± 0.014 |
| UTDHLTRI2 | 0.279 ± 0.013 | 0.255 ± 0.013 | 0.428 ± 0.014 |
| guir | 0.248 ± 0.012 | 0.247 ± 0.012 | 0.464 ± 0.013 |
| uog-linear-raw-expansion | 0.236 ± 0.013 | 0.228 ± 0.013 | 0.424 ± 0.016 |
| NYU-XL-f | 0.220 ± 0.012 | 0.206 ± 0.011 | 0.452 ± 0.014 |
| CUIS-F150 | 0.209 ± 0.010 | 0.172 ± 0.009 | 0.311 ± 0.013 |
| CUIS-MX5 | 0.195 ± 0.009 | 0.177 ± 0.009 | 0.422 ± 0.012 |
| NYU-L-f | 0.177 ± 0.011 | 0.156 ± 0.011 | 0.342 ± 0.016 |
| NYU-M-f | 0.176 ± 0.012 | 0.158 ± 0.011 | 0.333 ± 0.017 |
| DWS-UMA-SemqQueryExp | 0.173 ± 0.011 | 0.164 ± 0.011 | 0.328 ± 0.017 |
| DWS-UMA-SemqQueryExp20 | 0.170 ± 0.011 | 0.157 ± 0.011 | 0.318 ± 0.017 |
| DWS-UMA-SemqQueryExp30 | 0.162 ± 0.011 | 0.152 ± 0.011 | 0.311 ± 0.017 |
| CG-Seq2Seq | 0.054 ± 0.006 | 0.042 ± 0.005 | 0.083 ± 0.007 |
| Rprec/manual | map/manual | ndcg/manual | Rprec/automatic | map/automatic | ndcg/automatic | |
|---|---|---|---|---|---|---|
| UNH-e-L2R | 0.315 ± 0.013 | 0.310 ± 0.013 | 0.514 ± 0.015 | 0.181 ± 0.005 | 0.146 ± 0.005 | 0.316 ± 0.007 |
| UNH-e-mixed | 0.281 ± 0.013 | 0.266 ± 0.013 | 0.443 ± 0.015 | 0.176 ± 0.006 | 0.143 ± 0.005 | 0.300 ± 0.007 |
| UNH-e-graph | 0.276 ± 0.011 | 0.274 ± 0.011 | 0.476 ± 0.014 | 0.148 ± 0.005 | 0.114 ± 0.005 | 0.270 ± 0.006 |
| uog-linear-ltr-hier-ent | 0.266 ± 0.012 | 0.251 ± 0.012 | 0.430 ± 0.016 | 0.085 ± 0.004 | 0.066 ± 0.004 | 0.184 ± 0.005 |
| uog-paragraph-rf-ent | 0.262 ± 0.013 | 0.260 ± 0.013 | 0.486 ± 0.015 | 0.165 ± 0.005 | 0.144 ± 0.005 | 0.365 ± 0.007 |
| uog-heading-rh-sdm-ent | 0.235 ± 0.011 | 0.222 ± 0.011 | 0.398 ± 0.015 | 0.076 ± 0.004 | 0.060 ± 0.004 | 0.172 ± 0.005 |
| DWS-UMA-EntAspBM25none | 0.199 ± 0.012 | 0.184 ± 0.011 | 0.348 ± 0.015 | 0.063 ± 0.003 | 0.046 ± 0.003 | 0.153 ± 0.005 |
| DWS-UMA-EntAspQLrm | 0.194 ± 0.011 | 0.186 ± 0.011 | 0.357 ± 0.015 | 0.076 ± 0.004 | 0.055 ± 0.003 | 0.176 ± 0.005 |
| CUIS-dogeDodge | 0.131 ± 0.009 | 0.104 ± 0.008 | 0.231 ± 0.012 | 0.041 ± 0.002 | 0.025 ± 0.002 | 0.096 ± 0.004 |
| CUIS-Swift | 0.130 ± 0.009 | 0.103 ± 0.008 | 0.229 ± 0.012 | 0.041 ± 0.002 | 0.026 ± 0.002 | 0.096 ± 0.004 |
| CUIS-XTS | 0.126 ± 0.009 | 0.099 ± 0.008 | 0.211 ± 0.011 | 0.043 ± 0.002 | 0.026 ± 0.002 | 0.089 ± 0.004 |
Task: Passage Ranking
Run type: automatic
Run features:
Uses neural network technology
Uses other supervised machine learning method
Is trained with data provided as train-v1.5
Is trained with data provided as benchmarkY1train
Explanation of features:
none
Ranking methods used in pipeline
Deep Learning seq2seq, whoosh
Description of this run:
we developed a seq2seq model to map paragraphs to smaller set and then ranked the the set based on the similarity of query and the set.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1test automatic qrels
Ranking methods used in pipeline
1. BM25 2. SDM+WikiTreeLM
Description of this run:
sequential dependence model with restricted set of bigrams (it can be regarded as a special case of the "Latent Concept Expansion" model) WikiTreeLM (or generally WikiTreeSDM) selects a sequence of sections (i.e. a path in the wiki tree) of the corresponding Wikipedia article for each candidate paragraph, which maximizes a Dirchlet prior smoothed based scoring function. parameter tunning on benchmarkY1test automatic qrels
Citation of method:
"A Markov random field model for term dependencies", SIGIR 2005 "Latent Concept Expansion Using Markov Random Fields", SIGIR 2007 "CUIS Team for TREC 2017 CAR Track", TREC 2017
Judging precedence: Medium
Task: Passage Ranking
Run type: automatic
Run features:
Uses entity linking
Uses a dump of Wikipedia
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1test automatic qrels
Explanation of features:
the Wikipedia dump corresponds to DBpedia 2015-10
Ranking methods used in pipeline
1. BM25 2. SDM+WikiTreeLM+Entity LM
Description of this run:
SDM with restricted set of bigrams (it can be regarded as a special case of the "Latent Concept Expansion" model). WikiTreeLM (or generally, WikiTreeSDM) selects a sequence of sections of the corresponding Wikipedia article for each paragraph that maximizes a Dirchlet prior smoothed based scoring function. (consider weighted sum of concept frequencies/document lengths of each section) Entity LM%3A a simple entity language model. Use TAGME for entity linking on Wikipedia 2015-10. Identified query entities not in unprocessedAllButBenchmark are removed.
Citation of method:
"A Markov random field model for term dependencies", SIGIR 2005 "Latent Concept Expansion Using Markov Random Fields", SIGIR 2007 "CUIS Team for TREC 2017 CAR Track", TREC 2017 "Entity Retrieval in the Knowledge Graph with Hierarchical Entity Type and Content", ICTIR 2018
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses entity linking
Uses a dump of Wikipedia
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1test automatic qrels
Description of this run:
simply transform the passage ranking results in the run "CUIS-MX5"
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1test automatic qrels
Description of this run:
simply transform the passage ranking results in the run "CUIS-F150"
Judging precedence: Low
Task: Entity Ranking
Run type: automatic
Run features:
Uses entity linking
Uses a dump of Wikipedia
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1test automatic qrels
Description of this run:
simply transform the passage ranking results in the run "CUIS-MX5", complemented by LM on 'unprocessedAllButBenchmark'
Judging precedence: Medium
Task: Entity Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Ranking methods used in pipeline
bm25
Description of this run:
bm25
Citation of method:
Nanni, F., Ponzetto, S. P., & Dietz, L. (2018, May). Entity-Aspect Linking%3A Providing Fine-Grained Semantics of Entities in Context. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries (pp. 49-58). ACM.
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Ranking methods used in pipeline
bm25
Description of this run:
bm25
Citation of method:
Nanni, F., Ponzetto, S. P., & Dietz, L. (2018, May). Entity-Aspect Linking%3A Providing Fine-Grained Semantics of Entities in Context. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries (pp. 49-58). ACM.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Ranking methods used in pipeline
bm25
Description of this run:
Semantic Query Expansion with the k-nn of each query term + bm25 and boosting specific query terms, terms appearing on lower levels -> higher weights. We obtain the nearest neighbors from the GloVe Embedding Space. K=10
Citation of method:
Our method is a special version of [0]. If we take k=1 and replacing GloVe with a bilingual embedding space then you'd get the Term-by-Term Query Translation model from [0]. [0] https%3A//dl.acm.org/citation.cfm?id=3210157
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Ranking methods used in pipeline
bm25
Description of this run:
Semantic Query Expansion with the k-nn of each query term + bm25 and boosting specific query terms, terms appearing on lower levels -> higher weights. We obtain the nearest neighbors from the GloVe Embedding Space. K=20
Citation of method:
Our method is a special version of [0]. If we take k=1 and replacing GloVe with a bilingual embedding space then you'd get the Term-by-Term Query Translation model from [0]. [0] https%3A//dl.acm.org/citation.cfm?id=3210157
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Ranking methods used in pipeline
bm25
Description of this run:
Semantic Query Expansion with the k-nn of each query term + bm25 and boosting specific query terms, terms appearing on lower levels -> higher weights. We obtain the nearest neighbors from the GloVe Embedding Space. K=20
Citation of method:
Our method is a special version of [0]. If we take k=1 and replacing GloVe with a bilingual embedding space then you'd get the Term-by-Term Query Translation model from [0]. [0] https%3A//dl.acm.org/citation.cfm?id=3210157
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Uses neural network technology
Uses learning-to-rank
Is trained with data provided as train-v2.0
Is trained with data provided as benchmarkY1train
Is trained with data provided as benchmarkY1test automatic qrels
Explanation of features:
no external resources
Ranking methods used in pipeline
query likelihood
Description of this run:
PACRR neural ranking architecture modified with heading independence and heading frequency contextual vector
Citation of method:
@InProceedings{macavaney%3Asigir2018-car, author = {MacAvaney, Sean and Yates, Andrew and Cohan, Arman and Soldaini, Luca and Hui, Kai and Goharian, Nazli and Frieder, Ophir}, title = {Characterizing Question Facets for Complex Answer Retrieval}, booktitle = {Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval}, year = {2018}, url = {https%3A//arxiv.org/abs/1805.00791} }
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Uses neural network technology
Uses learning-to-rank
Is trained with data provided as train-v2.0
Is trained with data provided as benchmarkY1train
Is trained with data provided as benchmarkY1test automatic qrels
Explanation of features:
no external resources
Ranking methods used in pipeline
query likelihood, rm3 query expansion
Description of this run:
PACRR neural ranking architecture modified with heading independence, heading frequency contextual vector, and expanded query terms by heading
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses pre-trained word embeddings
Uses neural network technology
Uses learning-to-rank
Is trained with data provided as train-v2.0
Explanation of features:
We used train-v2.0 to train the reformulators and ranking model. Original Google's word2vec is used for word embeddings.
Ranking methods used in pipeline
The first ranked list is provided by BM25 (Lucene) then re-ranked using a simple neural network.
Description of this run:
Lucene is the underlying retrieval engine. We train 20 query reformulators from Nogueira and Cho (2017) on random disjoint partitions of the training set. For each query, each reformulator produces a list of ranked documents. We re-rank the union of these 20 lists using a simple ranking model that scores each query-document pair using a feed-forward neural network whose input is the concatenation of the average word embeddings of the query and document. To further improve the performance of the system, we train an ensemble of 10 ranking models whose network architectures are randomly chosen. For each query, we re-rank the union of the 10 lists produced by the 10 ranking models using the best ranking model in the ensemble.
Citation of method:
Nogueira, R. and Cho, K. (2017). Task-Oriented Query Reformulation with Reinforcement Learning. EMNLP.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses data v2.0, v1.5 or v1.4 data provided as 'unprocessedAllButBenchmark', 'unprocessedtrain',or 'halfwiki'
Uses pre-trained word embeddings
Uses neural network technology
Uses learning-to-rank
Explanation of features:
We used train-v2.0 to train the reformulators and ranking model. Original Google's word2vec is used for word embeddings.
Ranking methods used in pipeline
First ranked list provided by BM25 (Lucene) then re-ranked using a simple neural network.
Description of this run:
Lucene is the underlying retrieval engine. We train 20 query reformulators from Nogueira and Cho (2017) on random disjoint partitions of the training set. For each query, each reformulator produces a list of ranked documents. We re-rank the union of these 20 lists using a simple ranking model that scores each query-document pair using a feed-forward neural network whose input is the concatenation of the average word embeddings of the query and document. To further improve the performance of the system, we train an ensemble of 5 ranking models whose network architectures are randomly chosen. For each query, we re-rank the union of the 5 lists produced by the 5 ranking models using the best ranking model in the ensemble.
Citation of method:
Nogueira, R. and Cho, K. (2017). Task-Oriented Query Reformulation with Reinforcement Learning. EMNLP.
Judging precedence: Low
Task: Passage Ranking
Run type: automatic
Run features:
Uses data v2.0, v1.5 or v1.4 data provided as 'unprocessedAllButBenchmark', 'unprocessedtrain',or 'halfwiki'
Uses pre-trained word embeddings
Uses neural network technology
Uses learning-to-rank
Explanation of features:
We used train-v2.0 to train the reformulators and ranking model. Original Google's word2vec is used for word embeddings.
Ranking methods used in pipeline
First ranked list provided by BM25 (Lucene) then re-ranked using a simple neural network.
Description of this run:
Lucene is the underlying retrieval engine. We train 20 query reformulators from Nogueira and Cho (2017) on random disjoint partitions of the training set. For each query, each reformulator produces a list of ranked documents. We re-rank the union of these 20 lists using a simple ranking model that scores each query-document pair using a feed-forward neural network whose input is the concatenation of the average word embeddings of the query and document. To further improve the performance of the system, we train an ensemble of 11 ranking models whose network architectures are randomly chosen. For each query, we re-rank the union of the 11 lists produced by the 11 ranking models using the best ranking model in the ensemble.
Citation of method:
Nogueira, R. and Cho, K. (2017). Task-Oriented Query Reformulation with Reinforcement Learning. EMNLP.
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25, Query likelihood, Entity Context Model
Description of this run:
Four types of indices were made using Lucene v7.2.1%3A aspect, entity, page and paragraph.For each index, features were calculated using various combination of query-level,retrieval-model, expansion-model,and analyzer as follows%3A query-level%3A section path, all retrieval-model%3A BM25, Query Liklihood expansion-model%3A Entity Context Model analyzer%3A english Different indexes for retrieving entities and paragraphs are built from allButBenchmark and paragraphCorpus to obtain 10 rankings over Wikipages or paragraphs. Using the entity context model, we build entity relevance models (rankings over entity ids) from the pages/paragraphs, which are used as a ranking of entities. The particular run files are created by using a section path query to retrieve from these indices with BM25 and Querylikelihood. Additionally, paragraphs are retrieved by building a query from all headings of the topic page. These were then combined using a learning-to-rank and a combined ranking was obtained. This model is trained on benchmarkY1-train, using the document tree ground truth. The rankings were annotated with the highest ranked paragraph from an additional paragraph ranking as described below. PARAGRAPH FEATURES Fielded BM25%3A Weighted combination of unigram, bigram, and windowed bigram likelihood from the query to the document using BM25 algorithm. SDM%3A Standard SDM model Weighted Section Fielded BM25%3A Weighted combination of the leaf of a heading and the entire heading. Both scored with Fielded BM25. Expanded Bigrams%3A Using the rank score obtained from Fielded BM25, create a distribution over likely bigrams given unigrams in the query. Select the top 5 bigrams, add them to the original query, and rescore uses Fielded BM25. ENTITY -> PARAGRAPH FEATURES The following are entity features (scoring relevance of entity given query) that have been turned into paragraph features. Paragraphs are scored by integrating over the ranking scores of entities that they link to, given the entity feature. Link Freqeuncy%3A Each entity is scored according to the frequency that it is linked to by the candidate document (uses entity links). The following stats are parsed from v2.1 unprocessedAllButBenchmark%3A Categories%3A Unigram model of the enwiki categories of an entity. Entity Disambiguations%3A Unigram model of the disambiguation links of an entity Entity Inlinks%3A Unigram model of the disambiguation inlinks of an entity Entity Outlinks%3A Unigram model of the disambiguation outlinks of an entity Entity Redirects%3A Unigram model of the disambiguation redirect links of an entity Entity Sections%3A Unigram model of the section paths contained in the entity page. In addition, an "entity context" model is used. For each paragraph that an entity occurs (within the unprocessedAllButBenchmark v2.1), the surrounding unigrams, bigrams, and windowed bigrams are indexed. These -grams, along with the entity's name, are stored as pseudodocuments in an "entity context index" to be queried. Context Unigram%3A Query is tokenized into unigrams and used to query entity context database. Context Bigram%3A Query is tokenized into bigrams and used to query entity context database. Context Windowed Bigram%3A Query is tokenized into windowed bigram and used to query entity context database. TRAINING This model is trained on benchmarkY1-train, using both the document tree qrels and the entity tree qrels (because the document features can be transformed into entity features and vice versa, this allows us to use both training examples).
Judging precedence: Low
Task: Entity Ranking
Run type: automatic
Run features:
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25, Querylikelihood, RM3, ECM -- combined with unsupervised rank aggregation
Description of this run:
Trained on benchmarkt1train - tree.entity.qrels extended entity links with DBpedia Spotlight (retaining only links to entities that already have a link somewhere on the article) best variant selected on benchmarkY1test - extended tree.entity.qrels Different indexes for retrieving entities and paragraphs are built from allButBenchmark and paragraphCorpus. Rankings of entities provide a feature vector for nodes, Rankings of paragraphs provide a feature vector for edges. A specialized Learn-to-walk algorithm is used to train how node and edge features are best combined to obtain best degree centrality rankings. The Learn-to-walk is trained with mini-batched coordinate ascent to optimize MAP of entity rankings
Judging precedence: Medium
Task: Entity Ranking
Run type: automatic
Run features:
Uses entity linking
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25 using weighted combination of unigram, bigram, and windowed gram fields.
Description of this run:
Creating entity runs, then annotating entities with highest ranked paragraphs in a secondary ranking that contain this entity. Entity Ranking%3A ENTITY FEATURES Link Frequency%3A Each entity is scored according to the frequency that it is linked to by the candidate paragraphs (uses entity links). The following stats are parsed from v2.1 unprocessedAllButBenchmark%3A Categories%3A Unigram model of the enwiki categories of an entity. Entity Disambiguations%3A Unigram model of the disambiguation links of an entity Entity Inlinks%3A Unigram model of the disambiguation inlinks of an entity Entity Outlinks%3A Unigram model of the disambiguation outlinks of an entity Entity Redirects%3A Unigram model of the disambiguation redirect links of an entity Entity Sections%3A Unigram model of the section paths contained in the entity page. In addition, an "entity context" model is used. For each paragraph that an entity occurs (within the unprocessedAllButBenchmark v2.1), the surrounding unigrams, bigrams, and windowed bigrams are indexed. These -grams, along with the entity's name, are stored as pseudodocuments in an "entity context index" to be queried. Context Unigram%3A Query is tokenized into unigrams and used to query entity context database. Context Bigram%3A Query is tokenized into bigrams and used to query entity context database. Context Windowed Bigram%3A Query is tokenized into windowed bigram and used to query entity context database. PARAGRAPH -> ENTITY FEATURES The following are features that score the relevance of a paragraph given a query. These are turned into entity features by integrating over the scores of each paragraph that an entity links to. Fielded BM25%3A Weighted combination of unigram, bigram, and windowed bigram likelihood from the query to the document using BM25 algorithm. SDM%3A Standard SDM model Weighted Section Fielded BM25%3A Weighted combination of the leaf of a heading and the entire heading. Both scored with Fielded BM25. Expanded Bigrams%3A Using the rank score obtained from Fielded BM25, create a distribution over likely bigrams given unigrams in the query. Select the top 5 bigrams, add them to the original query, and rescore uses Fielded BM25. TRAINING This model is trained on benchmarkY1-train, using the document tree qrels. Paragraph Ranking%3A Fielded BM25%3A Weighted combination of unigram, bigram, and windowed bigram likelihood from the query to the document using BM25 algorithm. SDM%3A Standard SDM model Weighted Section Fielded BM25%3A Weighted combination of the leaf of a heading and the entire heading. Both scored with Fielded BM25. Expanded Bigrams%3A Using the rank score obtained from Fielded BM25, create a distribution over likely bigrams given unigrams in the query. Select the top 5 bigrams, add them to the original query, and rescore uses Fielded BM25. ENTITY -> PARAGRAPH FEATURES The following are entity features (scoring relevance of entity given query) that have been turned into paragraph features. Paragraphs are scored by integrating over the ranking scores of entities that they link to, given the entity feature. Link Freqeuncy%3A Each entity is scored according to the frequency that it is linked to by the candidate document (uses entity links). The following stats are parsed from v2.1 unprocessedAllButBenchmark%3A Categories%3A Unigram model of the enwiki categories of an entity. Entity Disambiguations%3A Unigram model of the disambiguation links of an entity Entity Inlinks%3A Unigram model of the disambiguation inlinks of an entity Entity Outlinks%3A Unigram model of the disambiguation outlinks of an entity Entity Redirects%3A Unigram model of the disambiguation redirect links of an entity Entity Sections%3A Unigram model of the section paths contained in the entity page. In addition, an "entity context" model is used. For each paragraph that an entity occurs (within the unprocessedAllButBenchmark v2.1), the surrounding unigrams, bigrams, and windowed bigrams are indexed. These -grams, along with the entity's name, are stored as pseudodocuments in an "entity context index" to be queried. Context Unigram%3A Query is tokenized into unigrams and used to query entity context database. Context Bigram%3A Query is tokenized into bigrams and used to query entity context database. Context Windowed Bigram%3A Query is tokenized into windowed bigram and used to query entity context database. TRAINING This model is trained on benchmarkY1-train, using both the document tree qrels and the entity tree qrels (because the document features can be transformed into entity features and vice versa, this allows us to use both training examples).
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25, RM3, Query likelihood
Description of this run:
Features%3A - BM25 (no expansion) - BM25 with RM3 expansion - Query likelihood (no expansion) - Query likelihood with RM3 expansion They are combined using learning to rank; RankLib with coordinate ascent optimized on map method is trained with data provided as benchmarkY1train External resources%3A Ranking%3A BM25, query likelihood (Lucene 7) Description%3A five fold cross validation is used with learning to rank to produce the training map. the full BY1train dataset is used with learning to rank to produce the runs on BY1test and BY2test. also in each run files used for combining using l2r, the retrieval scores are normalized which means for each query, scores are divided by the maximum score for that query.
Judging precedence: Low
Task: Passage Ranking
Run type: automatic
Run features:
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25 using weighted combination of unigram, bigram, and windowed gram fields.
Description of this run:
PARAGRAPH FEATURES Fielded BM25%3A Weighted combination of unigram, bigram, and windowed bigram likelihood from the query to the document using BM25 algorithm. SDM%3A Standard SDM model Weighted Section Fielded BM25%3A Weighted combination of the leaf of a heading and the entire heading. Both scored with Fielded BM25. Expanded Bigrams%3A Using the rank score obtained from Fielded BM25, create a distribution over likely bigrams given unigrams in the query. Select the top 5 bigrams, add them to the original query, and rescore uses Fielded BM25. ENTITY -> PARAGRAPH FEATURES The following are entity features (scoring relevance of entity given query) that have been turned into paragraph features. Paragraphs are scored by integrating over the ranking scores of entities that they link to, given the entity feature. Link Freqeuncy%3A Each entity is scored according to the frequency that it is linked to by the candidate document (uses entity links). The following stats are parsed from v2.1 unprocessedAllButBenchmark%3A Categories%3A Unigram model of the enwiki categories of an entity. Entity Disambiguations%3A Unigram model of the disambiguation links of an entity Entity Inlinks%3A Unigram model of the disambiguation inlinks of an entity Entity Outlinks%3A Unigram model of the disambiguation outlinks of an entity Entity Redirects%3A Unigram model of the disambiguation redirect links of an entity Entity Sections%3A Unigram model of the section paths contained in the entity page. In addition, an "entity context" model is used. For each paragraph that an entity occurs (within the unprocessedAllButBenchmark v2.1), the surrounding unigrams, bigrams, and windowed bigrams are indexed. These -grams, along with the entity's name, are stored as pseudodocuments in an "entity context index" to be queried. Context Unigram%3A Query is tokenized into unigrams and used to query entity context database. Context Bigram%3A Query is tokenized into bigrams and used to query entity context database. Context Windowed Bigram%3A Query is tokenized into windowed bigram and used to query entity context database. TRAINING This model is trained on benchmarkY1-train, using both the document tree qrels and the entity tree qrels (because the document features can be transformed into entity features and vice versa, this allows us to use both training examples).
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
None
Ranking methods used in pipeline
BM25 using weighted combination of unigram, bigram, and windowed gram fields.
Description of this run:
PARAGRAPH FEATURES Fielded BM25%3A Weighted combination of unigram, bigram, and windowed bigram likelihood from the query to the document using BM25 algorithm. TRAINING This model is trained on benchmarkY1-train, using the document tree qrels.
Judging precedence: Medium
Task: Passage Ranking
Run type: automatic
Run features:
Uses a knowledge graph such as DBpedia, Freebase, etc.
Uses entity linking
Uses learning-to-rank
Is trained with data provided as train-v2.0
Is trained with data provided as benchmarkY1train
Is trained with data provided as benchmarkY1test automatic qrels
Explanation of features:
Entity linking is used to find the queries in the entity. Dbpedia text is used to learn the entity embeddings with word2vec. learning to rank is used to find weights for the extracted features.
Ranking methods used in pipeline
rm3 sdm ql
Description of this run:
In this run, we want to investigate entity embeddings in the task of passage ranking. Each query and each document has an entity representation. Documents are represented as the average of entity embeddings within the passage. Queries have different representations including the complete outline and the subqueries. The similarity between query representations and document representation is used as features for learning to rank model.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Explanation of features:
None
Ranking methods used in pipeline
Sequential dependence model
Description of this run:
A baseline sequential dependence model run that uses the root and leaf heading, ignoring intermediate nodes.
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Is trained with data provided as benchmarkY1train
Ranking methods used in pipeline
Sequential dependence model
Description of this run:
A simple baseline run that uses SDM on the root and leaf heading.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses entity linking
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
It uses the SMAPH query annotator to annotate entity links in the query.
Ranking methods used in pipeline
Linear combination of features using the Ranklib coordinate ascent model
Description of this run:
A feature combination model that includes baseline retrieval methods (QL, SDM, RM3), as well as entity-based query expansion features from PRF on AllButBench and from query entity linking.
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses entity linking
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Uses learning-to-rank
Is trained with data provided as benchmarkY1train
Explanation of features:
It uses the SMAPH query annotator to annotate entity links in the query.
Ranking methods used in pipeline
Linear combination of features using the Ranklib coordinate ascent model.
Description of this run:
A feature combination model that includes baseline retrieval methods (QL, SDM, RM3), as well as entity-based query expansion features from PRF on AllButBench and from query entity linking.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses entity linking
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1train
Explanation of features:
SMAPH query annotations
Ranking methods used in pipeline
learning to rank model learned to weight expansion queries, mixture of QL and SDM, and entity expansion
Description of this run:
This is a full entity expansion run without a working set. It converts the learned expansion query weights into a galago weighted query and runs this against the full paragraph index.
Judging precedence: High
Task: Entity Ranking
Run type: automatic
Run features:
Uses entity linking
Uses data v2.1 data provided as 'unprocessedAllButBenchmark'
Is trained with data provided as benchmarkY1train
Explanation of features:
N/A
Ranking methods used in pipeline
Relevance model
Description of this run:
This is a entity relevance feedback run that is performed on top of uog-linear-ltr-hier.
Judging precedence: High
Task: Passage Ranking
Run type: automatic
Run features:
Uses data v2.0, v1.5 or v1.4 data provided as 'unprocessedAllButBenchmark', 'unprocessedtrain',or 'halfwiki'
Uses pre-trained word embeddings
Uses neural network technology
Is trained with data provided as train-v2.0
Explanation of features:
Pre-trained glove embeddings
Ranking methods used in pipeline
BM-25
Description of this run:
This method uses an initial BM-25 search followed by a neural re-ranking system using word embeddings, a cosine similarity matrix, and IR features.
Judging precedence: High