Evalbase Howto

Evalbase is a platform for managing the bookkeeping side of TREC and related evaluation workshops. It lets you register to participate, submit runs, and get evaluation results, as well as manage access for your participating organization.

Contents

• Getting Started
• Registering to participate
• Managing your organization
• The conference page
• Submitting runs
• Managing runs

Getting Started

You need to register an account on Evalbase. If you're reading this, that's probably already happened. Please make a personal account for you specifically, rather than a group account.

Once you're logged in, there is a drop-down menu in the upper right to view your profile or log out. The profile at the moment just contains contact information.

Register to participate

The main landing page for Evalbase (ir.nist.gov/evalbase/) lets you register to participate in any open evaluations. Clicking the 'Evalbase' title on any page will bring you back here. Typically that is just the current TREC cycle. Registering to participate involves creating an "organization" for your team.

When you register, you give the full name of your team ('Wossamatta University IR Group') and a short identifier ('wu_ir'). The short identifier gets used in a lot of places, for example in plots, so it should be short and sweet.

You also tick off which tracks you are interested in participating in. These choices don't lock you in, but we do use these choices to know which Slack channels to subscribe you to.

After creating your organization, there is a handy link you can give to members of your group to register themselves with the organization after they've set up their account. Click on your organization and the URL to share is there.

Managing your organization

From the main Evalbase page, you can click on your organization to see and edit details. The organization detail shows you your current members, the current track interests you checked, and the signup URL. If you want to add someone to your organization, ask them to create themselves an account, and then give them the signup URL.

You can also remove members from the organization from this page.

The conference page

Clicking on the name of the conference you've registered for brings you to the page for that conference. On this page, you can see all the tracks with links to their homepages. In each track there are one or more tasks you can submit runs to.

If a task is open for submissions, there is a link with a deadline listed. Your submitted runs are listed down the right hand side of the page, and by following those links you can edit or delete your submissions.

When the conference happens, evaluation summaries and plots are linked from this page as well.

Submitting runs

When a task is open for submission, there is a link on the conference page. Clicking that link will bring up a submission form that will ask you different questions about your run submission. These questions are used by the track coordinators and NIST to make comparisons between submissions.

The run identifier or "runtag" is a unique identifier for your run. Its common to use a common prefix for your runs based on your organization ID; if your organization ID was "NIST" you might tag your runs "NIST-1" or "NIST-firsttask-1". Using a common prefix makes it easy to spot your runs on results tables and plots with lots of runs on them.

After you click Submit, the run is uploaded and queued for validation. The validation process looks for errors in the submission file like missing topics or documents from the wrong collection. Validation can take a few seconds or a minute, and happens asynchronously, so check back in a few minutes to make sure your submission validated. If it didn't, you can see the output of the validator which should tell you where the issue is.

Evalbase sends an email to you and to the owner of your organization once validation is complete, so you'll have a notification of your submission and whether it validated or not.

Managing your submissions

Your submissions are listed on the right-hand side of the conference page. From that list, you can click a run to review and edit your answers to the submission form questions. You can also delete the run. Evalbase forces run identifiers to be unique, so if you want to re-submit the run, you need to delete it first.

Paper submissions for the proceedings are different in that run identifiers are generated automatically. In this case, you need to manually delete your old submissions if you submit a replacement.