Saw this error running the lastest version (0.0.3)

Traceback (most recent call last):
  File "src/feverous/baseline/predictor/train_verdict_predictor.py", line 19, in <module>
    from utils.annotation_processor import AnnotationProcessor, EvidenceType
ModuleNotFoundError: No module named 'utils'

This PR fixes the above path bugs. Now that the src code is being packed under a feverous folder, any global imports should start with "feverous." so that python can find the submodules etc.

There's a mix of global and relative path imports in the code base so I wasn't sure which one you wanted, I've gone with global imports for now since the scripts can still be run directly then

Note: This fix is required for the other PR (https://github.com/Raldir/FEVEROUS/pull/17) which I have put into draft pending this PR

When I run the steps in the README I get an error in one of the downstream steps due to missing an ID field. I did a basic grep through the code and found a commented out line that sets this. I re-added it and was able to run without issues subsequently.

https://github.com/Raldir/FEVEROUS/blob/bfbd67460bba9b4ff12c9901d76b70e34aaf1501/src/baseline/retriever/document_entity_tfidf_ir.py#L37

I think this needs to be uncommented?

Is it possible to provide the RoBERTa+NLI model as described in Table 4 of the paper? We'd like to compare with other methods other than NEI Sampling.

When I run the command PYTHONPATH=src/feverous python src/feverous/baseline/retriever/build_db.py --db_path data/feverous_wikiv1.db --save_path data/feverous-wiki-docs.db in README to reproduce the baseline, I encounter the following error message:

Since the GPL-licensed package `unidecode` is not installed, using Python's `unicodedata` package which yields worse results.
[INFO] 2022-02-21 20:06:52,952 - DrQA BuildDB - Reading into database...
Traceback (most recent call last):
  File "src/feverous/baseline/retriever/build_db.py", line 147, in <module>
    store_contents(
  File "src/feverous/baseline/retriever/build_db.py", line 102, in store_contents
    docs = db.get_doc_ids()
  File "/path/to/FEVEROUS/src/feverous/database/feverous_db.py", line 41, in get_doc_ids
    cursor.execute("SELECT id FROM wiki")
sqlite3.OperationalError: no such table: wiki

It seems that the data I downloaded is not complete. But the data is downloaded via ./scripts/download_data.sh, which should be OK.

Thanks for helping me!

Summary

• Adds instructions for running the README steps without conda (basic pip/python virtualenv)
• Adds a setup.py file to avoid having to set the PYTHONPATH manually before each command
• Does some linting on the README file to add new lines and code types to fenced code blocks

PS. Not sure if you want contributions at this point but I did this for myself when I was trying out the baseline and thought it might be helpful. Feel free to decline if not.

Note: to actually publish this on pypi you'll probably want to change the top-level namespace rather than installing from the src directory since the sub-package names are pretty general.

If you are open to changes I have a second PR in-place I can add later which adds a snakemake file so users can run 1 command for all the README steps

I was trying the feverous_scorer in evaluation and got confused about the format of "predicted_evidence" and "evidence".

1. In function "check_predicted_evidence_format",

assert all(len(prediction) == 3
                   for prediction in instance["predicted_evidence"]), \

indicates that the "predicted_evidence" is a set of evidence pieces in length of 3. But the required format of evidence in http://fever.ai is a list (at maximum three) of evidence sets. Should the "predicted_evidence" be flattened before processing feverous_scorer?

2. In function "is_strictly_correct",

for evience_group in instance["evidence"]:
       #Filter out the annotation ids. We just want the evidence page and line number
       actual_sentences = [[e[0], e[1], e[2]] for e in evience_group]
       #Only return true if an entire group of actual sentences is in the predicted sentences

       if all([actual_sent in instance["predicted_evidence"] for actual_sent in actual_sentences]):
           return True

indicates that the "actual_sentences" set contains evidence pieces in the format: (page,type,main_position). But the "evidence" in development set is a list of evidence sets, actual_sentences would not retrieve evidence pieces in the format: (page,type,main_position). Also, assume that actual_sentences contains evidence pieces in the format: (page,type,main_position), while the evidence pieces of "predicted_evidence" in the format: (page,type,position), it doesn't seem to return any true point.

Could you clarify the format of "predicted_evidence" and "evidence" for feverous_scorer?

I've noticed search queries with umlauts and other symbols also return nothing from the database while in fact there is a result in there.

For example: If I try to search for Didier Lourenço using db.get_doc_json() it returns nothing. But when I try to search the database directly using SELECT * FROM wiki WHERE id = 'Didier Lourenço'; it does return the correct result from the database.

The line under src/database/feverous_db.py to "normalize" the doc_id is commented out.

https://github.com/Raldir/FEVEROUS/blob/main/src/database/feverous_db.py#L47-L57

Using the "normalized" query seems to work. Any reason why this was commented out?

Hi, I have produced a prediction file and just wanted to obtain a result on the official evaluation website but found that did not work. Could you please point out any mistake in the submission format? Or is it just a BUG of this platform? The screenshot of my submission is shown below that it is always reminded that 'a valid publication URL is needed', where I am quite confused about.

Looking forward to your prompt reply.

@Raldir I am trying to run the fine-tuning training for the verdict prediction model but I keep running into CUDA memory issues. Do you remember what hardware specifications were required when you ran this?

When I run the command: PYTHONPATH=src python src/baseline/retriever/build_db.py --db_path data/feverous_wikiv1.db --save_path data/feverous-wiki-docs.db, I get the following error:

Traceback (most recent call last):
  File "src/baseline/retriever/build_db.py", line 19, in <module>
    from baseline.drqa.retriever import utils
  File "/home/martin/FEVEROUS/src/baseline/drqa/retriever/__init__.py", line 24, in <module>
    from .BM25_doc_ranker import BM25DocRanker
ModuleNotFoundError: No module named 'baseline.drqa.retriever.BM25_doc_ranker'

It seems that it is looking for the module BM25_doc_ranker, but this module does not seem to exist in the repo.

It seems that some documents have different IDs in the training data and in the database. For example the train data example nr 52, with the claim and evidence as follows:

'claim': 'Participating teams in the 2012–13 Macedonian First Football League '
          'included club Bregalnica, managed by Dobrinko Ilievski, and club '
          'Shkëndija, managed by Artim Shakiri, a retired football midfielder '
          'from North Macedonia.',
 'evidence': [{'content': ['2012–13 Macedonian First Football '
                           'League_cell_1_1_0',
                           '2012–13 Macedonian First Football '
                           'League_cell_1_1_1',
                           '2012–13 Macedonian First Football '
                           'League_cell_1_8_0',
                           '2012–13 Macedonian First Football '
                           'League_cell_1_8_1',
                           'Artim Šakiri_sentence_0'],

Here one part of the evidence is from the document Artim Šakiri. But trying to get a document with this ID from the database returns None. By looking through the list of IDs in the database I found that the closest one to Artim Šakiri is Artim Å akiri.

I am probably doing something incorrect here but I am not sure what. I got everything to run up to and including the combine script, src/baseline/retriever/combine_retrieval.py. I then downloaded the models linked in the README and tried to run the table cell extraction step and I run into this issue (see trace below)

/projects/creisle_prj/creisle_scratch/FEVEROUS/venv/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
[INFO] 2021-09-14 12:58:40,249 - LogHelper - Log Helper set up
[INFO] 2021-09-14 12:58:41,154 - __main__ - Start extracting cells from Tables...

  0%|                                                  | 0/7890 [00:00<?, ?it/s]
100%|███████████████████████████████████| 7890/7890 [00:00<00:00, 259469.96it/s]
Ignored unknown kwargs option trim_offsets
Traceback (most recent call last):
  File "src/baseline/retriever/predict_cells_from_table.py", line 322, in <module>
    main()
  File "src/baseline/retriever/predict_cells_from_table.py", line 317, in main
    extract_cells_from_tables(annotations, args)
  File "src/baseline/retriever/predict_cells_from_table.py", line 260, in extract_cells_from_tables
    predictions =  (model_output.predictions > 0.25).astype(int)
TypeError: '>' not supported between instances of 'NoneType' and 'float'

Have you seen this before? Any idea what I might have done wrong?

System specs OS: centos07 python: 3.7 conda or pip: pip

The claim of the seventh sample in dev set is "Per Axel Rydberg, born on July 6, 1860, in Odh, Västergötland, situated outside Sweden, was a graduate of University of Nebraska–Lincoln in the field of Botany (the science of plant life and a branch of biology).". I think this sample should be refused because Västergötland is in Sweden rather than outside Sweden. However, it is labeled "SUPPORTS". Does anyone have any ideas?