Evaluation

Evaluation is a process that takes a number of inputs/outputs pairs and aggregate them. You can always use the model directly and just parse its inputs/outputs manually to perform evaluation. Alternatively, evaluation is implemented in xmodaler.evaluation.

Custom Evaluators

X-modaler includes a few evaluators that compute metrics using standard dataset-specific APIs (e.g., COCO Captions). You can also implement your own evaluator that performs some other jobs using the inputs/outputs pairs:

from xmodaler.evaluation.build import EVALUATION_REGISTRY
@EVALUATION_REGISTRY.register()
class MyEvaler(object):
    def __init__(self, cfg, annfile):
        super(MyEvaler, self).__init__()
        ...

    def eval(self, result):
        ...

Use Evaluators

To evaluate using the methods of evaluators manually:

evaluator = MyEvaler(cfg, reference_file)
model.eval()
with torch.no_grad():
    outputs = model(inputs)
eval_results = evaluator.eval(outputs)