xmodaler.utils

xmodaler.utils.collect_env.collect_env_info()

xmodaler.utils.colormap.colormap(rgb=False, maximum=255)

Parameters:

• rgb (bool) – whether to return RGB colors or BGR colors.

• maximum (int) – either 255 or 1

Returns:

a float32 array of Nx3 colors, in range [0, 255] or [0, 1]

Return type:

ndarray

xmodaler.utils.colormap.random_color(rgb=False, maximum=255)

Parameters:

• rgb (bool) – whether to return RGB colors or BGR colors.

• maximum (int) – either 255 or 1

Returns:

a vector of 3 numbers

Return type:

ndarray

xmodaler.utils.comm.all_gather(data, group=None)

Run all_gather on arbitrary picklable data (not necessarily tensors).

Parameters:

• data – any picklable object

• group – a torch process group. By default, will use a group which contains all ranks on gloo backend.

Returns:

list of data gathered from each rank

Return type:

list[data]

xmodaler.utils.comm.gather(data, dst=0, group=None)

Run gather on arbitrary picklable data (not necessarily tensors).

Parameters:

• data – any picklable object

• dst (int) – destination rank

• group – a torch process group. By default, will use a group which contains all ranks on gloo backend.

Returns:

on dst, a list of data gathered from each rank. Otherwise,

an empty list.

Return type:

list[data]

xmodaler.utils.distributed.all_gather_list(data)

Gathers arbitrary data from all nodes into a list.

. autofunction:: xmodaler.utils.comm.any_broadcast

xmodaler.utils.comm.get_local_rank() → int

Returns:

The rank of the current process within the local (per-machine) process group.

xmodaler.utils.comm.get_local_size() → int

Returns:

The size of the per-machine process group, i.e. the number of processes per machine.

xmodaler.utils.comm.get_rank() → int

xmodaler.utils.comm.get_world_size() → int

xmodaler.utils.comm.is_main_process() → bool

xmodaler.utils.comm.reduce_dict(input_dict, average=True)

Reduce the values in the dictionary from all processes so that process with rank 0 has the reduced results.

Parameters:

• input_dict (dict) – inputs to be reduced. All the values must be scalar CUDA Tensor.

• average (bool) – whether to do average or sum

Returns:

a dict with the same keys as input_dict, after reduction.

xmodaler.utils.comm.shared_random_seed()

Returns:

a random number that is the same across all workers.

If workers need a shared RNG, they can use this shared seed to create one.

Return type:

int

All workers must call this function, otherwise it will deadlock.

xmodaler.utils.comm.synchronize()

Helper function to synchronize (barrier) among all processes when using distributed training

xmodaler.utils.comm.unwrap_model(model)

xmodaler.utils.env.seed_all_rng(seed=None)

Set the random seed for the RNG in torch, numpy and python.

Parameters:

seed (int) – if None, will use a strong random seed.

xmodaler.utils.env.setup_environment()

Perform environment setup work. The default setup is a no-op, but this function allows the user to specify a Python source file or a module in the $DETECTRON2_ENV_MODULE environment variable, that performs custom setup work that may be necessary to their computing environment.

xmodaler.utils.env.setup_custom_environment(custom_module)

Load custom environment setup by importing a Python source file or a module, and run the setup function.

xmodaler.utils.events.get_event_storage()

Returns:

The EventStorage object that’s currently being used. Throws an error if no EventStorage is currently enabled.

class xmodaler.utils.events.JSONWriter(json_file, window_size=20)

Bases: EventWriter

Write scalars to a json file.

It saves scalars as one json per line (instead of a big json) for easy parsing.

Examples parsing such a json file:

$ cat metrics.json | jq -s '.[0:2]'
[
  {
    "data_time": 0.008433341979980469,
    "iteration": 19,
    "loss": 1.9228371381759644,
    "loss_box_reg": 0.050025828182697296,
    "loss_classifier": 0.5316952466964722,
    "loss_mask": 0.7236229181289673,
    "loss_rpn_box": 0.0856662318110466,
    "loss_rpn_cls": 0.48198649287223816,
    "lr": 0.007173333333333333,
    "time": 0.25401854515075684
  },
  {
    "data_time": 0.007216215133666992,
    "iteration": 39,
    "loss": 1.282649278640747,
    "loss_box_reg": 0.06222952902317047,
    "loss_classifier": 0.30682939291000366,
    "loss_mask": 0.6970193982124329,
    "loss_rpn_box": 0.038663312792778015,
    "loss_rpn_cls": 0.1471673548221588,
    "lr": 0.007706666666666667,
    "time": 0.2490077018737793
  }
]

$ cat metrics.json | jq '.loss_mask'
0.7126231789588928
0.689423680305481
0.6776131987571716
...

__init__(json_file, window_size=20)

Parameters:

• json_file (str) – path to the json file. New data will be appended if the file exists.

• window_size (int) – the window size of median smoothing for the scalars whose smoothing_hint are True.

close()

write()

class xmodaler.utils.events.TensorboardXWriter(log_dir: str, window_size: int = 20, **kwargs)

Bases: EventWriter

Write all scalars to a tensorboard file.

__init__(log_dir: str, window_size: int = 20, **kwargs)

Parameters:

• log_dir (str) – the directory to save the output events

• window_size (int) – the scalars will be median-smoothed by this window size

• kwargs – other arguments passed to torch.utils.tensorboard.SummaryWriter(…)

close()

write()

class xmodaler.utils.events.CommonMetricPrinter(max_iter: Optional[int] = None)

Bases: EventWriter

Print common metrics to the terminal, including iteration time, ETA, memory, all losses, and the learning rate. It also applies smoothing using a window of 20 elements.

It’s meant to print common metrics in common ways. To print something in more customized ways, please implement a similar printer by yourself.

__init__(max_iter: Optional[int] = None)

Parameters:

max_iter – the maximum number of iterations to train. Used to compute ETA. If not given, ETA will not be printed.

write()

class xmodaler.utils.events.EventStorage(start_iter=0)

Bases: object

The user-facing class that provides metric storage functionalities.

In the future we may add support for storing / logging other types of data if needed.

__init__(start_iter=0)

Parameters:

start_iter (int) – the iteration number to start with

clear_histograms()

Delete all the stored histograms for visualization. This should be called after histograms are written to tensorboard.

clear_images()

Delete all the stored images for visualization. This should be called after images are written to tensorboard.

histories()

Returns:

the HistoryBuffer for all scalars

Return type:

dict[name -> HistoryBuffer]

history(name)

Returns:

the scalar history for name

Return type:

HistoryBuffer

property iter

Returns: int: The current iteration number. When used together with a trainer,

this is ensured to be the same as trainer.iter.

property iteration

latest()

Returns:

mapping from the name of each scalar to the most

recent value and the iteration number its added.

Return type:

dict[str -> (float, int)]

latest_with_smoothing_hint(window_size=20)

Similar to latest(), but the returned values are either the un-smoothed original latest value, or a median of the given window_size, depend on whether the smoothing_hint is True.

This provides a default behavior that other writers can use.

name_scope(name)

Yields:

A context within which all the events added to this storage will be prefixed by the name scope.

put_histogram(hist_name, hist_tensor, bins=1000)

Create a histogram from a tensor.

Parameters:

• hist_name (str) – The name of the histogram to put into tensorboard.

• hist_tensor (torch.Tensor) – A Tensor of arbitrary shape to be converted into a histogram.

• bins (int) – Number of histogram bins.

put_image(img_name, img_tensor)

Add an img_tensor associated with img_name, to be shown on tensorboard.

Parameters:

• img_name (str) – The name of the image to put into tensorboard.

• img_tensor (torch.Tensor or numpy.array) – An uint8 or float Tensor of shape [channel, height, width] where channel is 3. The image format should be RGB. The elements in img_tensor can either have values in [0, 1] (float32) or [0, 255] (uint8). The img_tensor will be visualized in tensorboard.

put_scalar(name, value, smoothing_hint=True)

Add a scalar value to the HistoryBuffer associated with name.

Parameters:

smoothing_hint (bool) –

a ‘hint’ on whether this scalar is noisy and should be smoothed when logged. The hint will be accessible through EventStorage.smoothing_hints(). A writer may ignore the hint and apply custom smoothing rule.

It defaults to True because most scalars we save need to be smoothed to provide any useful signal.

put_scalars(*, smoothing_hint=True, **kwargs)

Put multiple scalars from keyword arguments.

Examples

storage.put_scalars(loss=my_loss, accuracy=my_accuracy, smoothing_hint=True)

smoothing_hints()

Returns:

the user-provided hint on whether the scalar

is noisy and needs smoothing.

Return type:

dict[name -> bool]

step()

User should either: (1) Call this function to increment storage.iter when needed. Or (2) Set storage.iter to the correct iteration number before each iteration.

The storage will then be able to associate the new data with an iteration number.

class xmodaler.utils.file_io.PathHandler(async_executor: Optional[Executor] = None)

Bases: EventLogger

PathHandler is a base class that defines common I/O functionality for a URI protocol. It routes I/O for a generic URI which may look like “protocol://*” or a canonical filepath “/foo/bar/baz”.

__init__(async_executor: Optional[Executor] = None) → None

When registering a PathHandler, the user can optionally pass in a Executor to run the asynchronous file operations. NOTE: For regular non-async operations of PathManager, there is no need to pass async_executor.

Parameters:

async_executor (optional Executor) –

Used for async file operations. Usage: ```

path_handler = NativePathHandler(async_executor=exe) path_manager.register_handler(path_handler)

```

xmodaler.utils.file_io.PathManager

This is a detectron2 project-specific PathManager. We try to stay away from global PathManager in fvcore as it introduces potential conflicts among other libraries.

class xmodaler.utils.logger.Counter(**kwds)

Bases: dict

Dict subclass for counting hashable items. Sometimes called a bag or multiset. Elements are stored as dictionary keys and their counts are stored as dictionary values.

>>> c = Counter('abcdeabcdabcaba')  # count elements from a string

>>> c.most_common(3)                # three most common elements
[('a', 5), ('b', 4), ('c', 3)]
>>> sorted(c)                       # list all unique elements
['a', 'b', 'c', 'd', 'e']
>>> ''.join(sorted(c.elements()))   # list elements with repetitions
'aaaaabbbbcccdde'
>>> sum(c.values())                 # total of all counts
15

>>> c['a']                          # count of letter 'a'
5
>>> for elem in 'shazam':           # update counts from an iterable
...     c[elem] += 1                # by adding 1 to each element's count
>>> c['a']                          # now there are seven 'a'
7
>>> del c['b']                      # remove all 'b'
>>> c['b']                          # now there are zero 'b'
0

>>> d = Counter('simsalabim')       # make another counter
>>> c.update(d)                     # add in the second counter
>>> c['a']                          # now there are nine 'a'
9

>>> c.clear()                       # empty the counter
>>> c
Counter()

Note: If a count is set to zero or reduced to zero, it will remain in the counter until the entry is deleted or the counter is cleared:

>>> c = Counter('aaabbc')
>>> c['b'] -= 2                     # reduce the count of 'b' by two
>>> c.most_common()                 # 'b' is still in, but its count is zero
[('a', 3), ('c', 1), ('b', 0)]

__init__(**kwds)

Create a new, empty Counter object. And if given, count elements from an input iterable. Or, initialize the count from another mapping of elements to their counts.

>>> c = Counter()                           # a new, empty counter
>>> c = Counter('gallahad')                 # a new counter from an iterable
>>> c = Counter({'a': 4, 'b': 2})           # a new counter from a mapping
>>> c = Counter(a=4, b=2)                   # a new counter from keyword args

copy()

Return a shallow copy.

elements()

Iterator over elements repeating each as many times as its count.

>>> c = Counter('ABCABC')
>>> sorted(c.elements())
['A', 'A', 'B', 'B', 'C', 'C']

# Knuth’s example for prime factors of 1836: 2**2 * 3**3 * 17**1 >>> prime_factors = Counter({2: 2, 3: 3, 17: 1}) >>> product = 1 >>> for factor in prime_factors.elements(): # loop over factors … product *= factor # and multiply them >>> product 1836

Note, if an element’s count has been set to zero or is a negative number, elements() will ignore it.

classmethod fromkeys(iterable, v=None)

Create a new dictionary with keys from iterable and values set to value.

most_common(n=None)

List the n most common elements and their counts from the most common to the least. If n is None, then list all element counts.

>>> Counter('abcdeabcdabcaba').most_common(3)
[('a', 5), ('b', 4), ('c', 3)]

subtract(**kwds)

Like dict.update() but subtracts counts instead of replacing them. Counts can be reduced below zero. Both the inputs and outputs are allowed to contain zero and negative counts.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.subtract('witch')             # subtract elements from another iterable
>>> c.subtract(Counter('watch'))    # subtract elements from another counter
>>> c['h']                          # 2 in which, minus 1 in witch, minus 1 in watch
0
>>> c['w']                          # 1 in which, minus 1 in witch, minus 1 in watch
-1

update(**kwds)

Like dict.update() but add counts instead of replacing them.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.update('witch')           # add elements from another iterable
>>> d = Counter('watch')
>>> c.update(d)                 # add elements from another counter
>>> c['h']                      # four 'h' in which, witch, and watch
4

xmodaler.utils.logger.colored(text: str, color: str | None = None, on_color: str | None = None, attrs: Iterable[str] | None = None) → str

Colorize text.

Available text colors:

black, red, green, yellow, blue, magenta, cyan, white, light_grey, dark_grey, light_red, light_green, light_yellow, light_blue, light_magenta, light_cyan.

Available text highlights:

on_black, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, on_white, on_light_grey, on_dark_grey, on_light_red, on_light_green, on_light_yellow, on_light_blue, on_light_magenta, on_light_cyan.

Available attributes:

bold, dark, underline, blink, reverse, concealed.

Example

colored(‘Hello, World!’, ‘red’, ‘on_black’, [‘bold’, ‘blink’]) colored(‘Hello, World!’, ‘green’)

xmodaler.utils.logger.create_small_table(small_dict)

Create a small table using the keys of small_dict as headers. This is only suitable for small dictionaries.

Parameters:

small_dict (dict) – a result dictionary of only a few items.

Returns:

the table as a string.

Return type:

str

xmodaler.utils.logger.log_every_n(lvl, msg, n=1, *, name=None)

Log once per n times.

Parameters:

• lvl (int) – the logging level

• msg (str) –

• n (int) –

• name (str) – name of the logger to use. Will use the caller’s module by default.

xmodaler.utils.logger.log_every_n_seconds(lvl, msg, n=1, *, name=None)

Log no more than once per n seconds.

Parameters:

• lvl (int) – the logging level

• msg (str) –

• n (int) –

• name (str) – name of the logger to use. Will use the caller’s module by default.

xmodaler.utils.logger.log_first_n(lvl, msg, n=1, *, name=None, key='caller')

Log only for the first n times.

Parameters:

• lvl (int) – the logging level

• msg (str) –

• n (int) –

• name (str) – name of the logger to use. Will use the caller’s module by default.

• key (str or tuple[str]) – the string(s) can be one of “caller” or “message”, which defines how to identify duplicated logs. For example, if called with n=1, key=”caller”, this function will only log the first call from the same caller, regardless of the message content. If called with n=1, key=”message”, this function will log the same content only once, even if they are called from different places. If called with n=1, key=(“caller”, “message”), this function will not log only if the same caller has logged the same message before.

xmodaler.utils.logger.setup_logger(output=None, distributed_rank=0, *, color=True, name='xmodaler', abbrev_name=None)

Initialize the xmodaler logger and set its verbosity level to “DEBUG”.

Parameters:

• output (str) – a file name or a directory to save log. If None, will not save log file. If ends with “.txt” or “.log”, assumed to be a file name. Otherwise, logs will be saved to output/log.txt.

• name (str) – the root module name of this logger

• abbrev_name (str) – an abbreviation of the module, to avoid long names in logs. Set to “” to not log the root module in logs. By default, will abbreviate “xmodaler” to “d2” and leave other modules unchanged.

Returns:

a logger

Return type:

logging.Logger

xmodaler.utils.logger.tabulate(tabular_data, headers=(), tablefmt='simple', floatfmt='g', intfmt='', numalign='default', stralign='default', missingval='', showindex='default', disable_numparse=False, colalign=None, maxcolwidths=None, rowalign=None, maxheadercolwidths=None)

Format a fixed width table for pretty printing.

>>> print(tabulate([[1, 2.34], [-56, "8.999"], ["2", "10001"]]))
---  ---------
  1      2.34
-56      8.999
  2  10001
---  ---------

The first required argument (tabular_data) can be a list-of-lists (or another iterable of iterables), a list of named tuples, a dictionary of iterables, an iterable of dictionaries, an iterable of dataclasses (Python 3.7+), a two-dimensional NumPy array, NumPy record array, or a Pandas’ dataframe.

Table headers

To print nice column headers, supply the second argument (headers):

• headers can be an explicit list of column headers

• if headers=”firstrow”, then the first row of data is used

• if headers=”keys”, then dictionary keys or column indices are used

Otherwise a headerless table is produced.

If the number of headers is less than the number of columns, they are supposed to be names of the last columns. This is consistent with the plain-text format of R and Pandas’ dataframes.

>>> print(tabulate([["sex","age"],["Alice","F",24],["Bob","M",19]],
...       headers="firstrow"))
       sex      age
-----  -----  -----
Alice  F         24
Bob    M         19

By default, pandas.DataFrame data have an additional column called row index. To add a similar column to all other types of data, use showindex=”always” or showindex=True. To suppress row indices for all types of data, pass showindex=”never” or `showindex=False. To add a custom row index column, pass showindex=some_iterable.

>>> print(tabulate([["F",24],["M",19]], showindex="always"))
-  -  --
0  F  24
1  M  19
-  -  --

Column alignment

tabulate tries to detect column types automatically, and aligns the values properly. By default it aligns decimal points of the numbers (or flushes integer numbers to the right), and flushes everything else to the left. Possible column alignments (numalign, stralign) are: “right”, “center”, “left”, “decimal” (only for numalign), and None (to disable alignment).

Table formats

intfmt is a format specification used for columns which contain numeric data without a decimal point. This can also be a list or tuple of format strings, one per column.

floatfmt is a format specification used for columns which contain numeric data with a decimal point. This can also be a list or tuple of format strings, one per column.

None values are replaced with a missingval string (like floatfmt, this can also be a list of values for different columns):

>>> print(tabulate([["spam", 1, None],
...                 ["eggs", 42, 3.14],
...                 ["other", None, 2.7]], missingval="?"))
-----  --  ----
spam    1  ?
eggs   42  3.14
other   ?  2.7
-----  --  ----

Various plain-text table formats (tablefmt) are supported: ‘plain’, ‘simple’, ‘grid’, ‘pipe’, ‘orgtbl’, ‘rst’, ‘mediawiki’, ‘latex’, ‘latex_raw’, ‘latex_booktabs’, ‘latex_longtable’ and tsv. Variable `tabulate_formats`contains the list of currently supported formats.

“plain” format doesn’t use any pseudographics to draw tables, it separates columns with a double space:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                 ["strings", "numbers"], "plain"))
strings      numbers
spam         41.9999
eggs        451

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="plain"))
spam   41.9999
eggs  451

“simple” format is like Pandoc simple_tables:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                 ["strings", "numbers"], "simple"))
strings      numbers
---------  ---------
spam         41.9999
eggs        451

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="simple"))
----  --------
spam   41.9999
eggs  451
----  --------

“grid” is similar to tables produced by Emacs table.el package or Pandoc grid_tables:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "grid"))
+-----------+-----------+
| strings   |   numbers |
+===========+===========+
| spam      |   41.9999 |
+-----------+-----------+
| eggs      |  451      |
+-----------+-----------+

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="grid"))
+------+----------+
| spam |  41.9999 |
+------+----------+
| eggs | 451      |
+------+----------+

“simple_grid” draws a grid using single-line box-drawing characters:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "simple_grid"))
┌───────────┬───────────┐
│ strings   │   numbers │
├───────────┼───────────┤
│ spam      │   41.9999 │
├───────────┼───────────┤
│ eggs      │  451      │
└───────────┴───────────┘

“rounded_grid” draws a grid using single-line box-drawing characters with rounded corners:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "rounded_grid"))
╭───────────┬───────────╮
│ strings   │   numbers │
├───────────┼───────────┤
│ spam      │   41.9999 │
├───────────┼───────────┤
│ eggs      │  451      │
╰───────────┴───────────╯

“heavy_grid” draws a grid using bold (thick) single-line box-drawing characters:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "heavy_grid"))
┏━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ strings   ┃   numbers ┃
┣━━━━━━━━━━━╋━━━━━━━━━━━┫
┃ spam      ┃   41.9999 ┃
┣━━━━━━━━━━━╋━━━━━━━━━━━┫
┃ eggs      ┃  451      ┃
┗━━━━━━━━━━━┻━━━━━━━━━━━┛

“mixed_grid” draws a grid using a mix of light (thin) and heavy (thick) lines box-drawing characters:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "mixed_grid"))
┍━━━━━━━━━━━┯━━━━━━━━━━━┑
│ strings   │   numbers │
┝━━━━━━━━━━━┿━━━━━━━━━━━┥
│ spam      │   41.9999 │
├───────────┼───────────┤
│ eggs      │  451      │
┕━━━━━━━━━━━┷━━━━━━━━━━━┙

“double_grid” draws a grid using double-line box-drawing characters:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "double_grid"))
╔═══════════╦═══════════╗
║ strings   ║   numbers ║
╠═══════════╬═══════════╣
║ spam      ║   41.9999 ║
╠═══════════╬═══════════╣
║ eggs      ║  451      ║
╚═══════════╩═══════════╝

“fancy_grid” draws a grid using a mix of single and double-line box-drawing characters:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "fancy_grid"))
╒═══════════╤═══════════╕
│ strings   │   numbers │
╞═══════════╪═══════════╡
│ spam      │   41.9999 │
├───────────┼───────────┤
│ eggs      │  451      │
╘═══════════╧═══════════╛

“outline” is the same as the “grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "outline"))
+-----------+-----------+
| strings   |   numbers |
+===========+===========+
| spam      |   41.9999 |
| eggs      |  451      |
+-----------+-----------+

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="outline"))
+------+----------+
| spam |  41.9999 |
| eggs | 451      |
+------+----------+

“simple_outline” is the same as the “simple_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "simple_outline"))
┌───────────┬───────────┐
│ strings   │   numbers │
├───────────┼───────────┤
│ spam      │   41.9999 │
│ eggs      │  451      │
└───────────┴───────────┘

“rounded_outline” is the same as the “rounded_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "rounded_outline"))
╭───────────┬───────────╮
│ strings   │   numbers │
├───────────┼───────────┤
│ spam      │   41.9999 │
│ eggs      │  451      │
╰───────────┴───────────╯

“heavy_outline” is the same as the “heavy_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "heavy_outline"))
┏━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ strings   ┃   numbers ┃
┣━━━━━━━━━━━╋━━━━━━━━━━━┫
┃ spam      ┃   41.9999 ┃
┃ eggs      ┃  451      ┃
┗━━━━━━━━━━━┻━━━━━━━━━━━┛

“mixed_outline” is the same as the “mixed_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "mixed_outline"))
┍━━━━━━━━━━━┯━━━━━━━━━━━┑
│ strings   │   numbers │
┝━━━━━━━━━━━┿━━━━━━━━━━━┥
│ spam      │   41.9999 │
│ eggs      │  451      │
┕━━━━━━━━━━━┷━━━━━━━━━━━┙

“double_outline” is the same as the “double_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "double_outline"))
╔═══════════╦═══════════╗
║ strings   ║   numbers ║
╠═══════════╬═══════════╣
║ spam      ║   41.9999 ║
║ eggs      ║  451      ║
╚═══════════╩═══════════╝

“fancy_outline” is the same as the “fancy_grid” format but doesn’t draw lines between rows:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "fancy_outline"))
╒═══════════╤═══════════╕
│ strings   │   numbers │
╞═══════════╪═══════════╡
│ spam      │   41.9999 │
│ eggs      │  451      │
╘═══════════╧═══════════╛

“pipe” is like tables in PHP Markdown Extra extension or Pandoc pipe_tables:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "pipe"))
| strings   |   numbers |
|:----------|----------:|
| spam      |   41.9999 |
| eggs      |  451      |

“presto” is like tables produce by the Presto CLI:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "presto"))
 strings   |   numbers
-----------+-----------
 spam      |   41.9999
 eggs      |  451

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="pipe"))
|:-----|---------:|
| spam |  41.9999 |
| eggs | 451      |

“orgtbl” is like tables in Emacs org-mode and orgtbl-mode. They are slightly different from “pipe” format by not using colons to define column alignment, and using a “+” sign to indicate line intersections:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "orgtbl"))
| strings   |   numbers |
|-----------+-----------|
| spam      |   41.9999 |
| eggs      |  451      |

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="orgtbl"))
| spam |  41.9999 |
| eggs | 451      |

“rst” is like a simple table format from reStructuredText; please note that reStructuredText accepts also “grid” tables:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]],
...                ["strings", "numbers"], "rst"))
=========  =========
strings      numbers
=========  =========
spam         41.9999
eggs        451
=========  =========

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="rst"))
====  ========
spam   41.9999
eggs  451
====  ========

“mediawiki” produces a table markup used in Wikipedia and on other MediaWiki-based sites:

>>> print(tabulate([["strings", "numbers"], ["spam", 41.9999], ["eggs", "451.0"]],
...                headers="firstrow", tablefmt="mediawiki"))
{| class="wikitable" style="text-align: left;"
|+ <!-- caption -->
|-
! strings   !! align="right"|   numbers
|-
| spam      || align="right"|   41.9999
|-
| eggs      || align="right"|  451
|}

“html” produces HTML markup as an html.escape’d str with a ._repr_html_ method so that Jupyter Lab and Notebook display the HTML and a .str property so that the raw HTML remains accessible the unsafehtml table format can be used if an unescaped HTML format is required:

>>> print(tabulate([["strings", "numbers"], ["spam", 41.9999], ["eggs", "451.0"]],
...                headers="firstrow", tablefmt="html"))
<table>
<thead>
<tr><th>strings  </th><th style="text-align: right;">  numbers</th></tr>
</thead>
<tbody>
<tr><td>spam     </td><td style="text-align: right;">  41.9999</td></tr>
<tr><td>eggs     </td><td style="text-align: right;"> 451     </td></tr>
</tbody>
</table>

“latex” produces a tabular environment of LaTeX document markup:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="latex"))
\begin{tabular}{lr}
\hline
 spam &  41.9999 \\
 eggs & 451      \\
\hline
\end{tabular}

“latex_raw” is similar to “latex”, but doesn’t escape special characters, such as backslash and underscore, so LaTeX commands may embedded into cells’ values:

>>> print(tabulate([["spam$_9$", 41.9999], ["\\emph{eggs}", "451.0"]], tablefmt="latex_raw"))
\begin{tabular}{lr}
\hline
 spam$_9$    &  41.9999 \\
 \emph{eggs} & 451      \\
\hline
\end{tabular}

“latex_booktabs” produces a tabular environment of LaTeX document markup using the booktabs.sty package:

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="latex_booktabs"))
\begin{tabular}{lr}
\toprule
 spam &  41.9999 \\
 eggs & 451      \\
\bottomrule
\end{tabular}

“latex_longtable” produces a tabular environment that can stretch along multiple pages, using the longtable package for LaTeX.

>>> print(tabulate([["spam", 41.9999], ["eggs", "451.0"]], tablefmt="latex_longtable"))
\begin{longtable}{lr}
\hline
 spam &  41.9999 \\
 eggs & 451      \\
\hline
\end{longtable}

Number parsing

By default, anything which can be parsed as a number is a number. This ensures numbers represented as strings are aligned properly. This can lead to weird results for particular strings such as specific git SHAs e.g. “42992e1” will be parsed into the number 429920 and aligned as such.

To completely disable number parsing (and alignment), use disable_numparse=True. For more fine grained control, a list column indices is used to disable number parsing only on those columns e.g. disable_numparse=[0, 2] would disable number parsing only on the first and third columns.

Column Widths and Auto Line Wrapping

Tabulate will, by default, set the width of each column to the length of the longest element in that column. However, in situations where fields are expected to reasonably be too long to look good as a single line, tabulate can help automate word wrapping long fields for you. Use the parameter maxcolwidth to provide a list of maximal column widths

>>> print(tabulate(           [('1', 'John Smith',             'This is a rather long description that might look better if it is wrapped a bit')],           headers=("Issue Id", "Author", "Description"),           maxcolwidths=[None, None, 30],           tablefmt="grid"          ))
+------------+------------+-------------------------------+
|   Issue Id | Author     | Description                   |
+============+============+===============================+
|          1 | John Smith | This is a rather long         |
|            |            | description that might look   |
|            |            | better if it is wrapped a bit |
+------------+------------+-------------------------------+

Header column width can be specified in a similar way using maxheadercolwidth

xmodaler.utils.memory.retry_if_cuda_oom(func)

Makes a function retry itself after encountering pytorch’s CUDA OOM error. It will first retry after calling torch.cuda.empty_cache().

If that still fails, it will then retry by trying to convert inputs to CPUs. In this case, it expects the function to dispatch to CPU implementation. The return values may become CPU tensors as well and it’s user’s responsibility to convert it back to CUDA tensor if needed.

Parameters:

func – a stateless callable that takes tensor-like objects as arguments

Returns:

a callable which retries func if OOM is encountered.

Examples:

output = retry_if_cuda_oom(some_torch_function)(input1, input2)
# output may be on CPU even if inputs are on GPU

Note

1. When converting inputs to CPU, it will only look at each argument and check if it has .device and .to for conversion. Nested structures of tensors are not supported.

2. Since the function might be called more than once, it has to be stateless.

class xmodaler.utils.registry.Registry(name: str)

Bases: Iterable[Tuple[str, Any]]

The registry that provides name -> object mapping, to support third-party users’ custom modules.

To create a registry (e.g. a backbone registry):

BACKBONE_REGISTRY = Registry('BACKBONE')

To register an object:

@BACKBONE_REGISTRY.register()
class MyBackbone():
    ...

Or:

BACKBONE_REGISTRY.register(MyBackbone)

__init__(name: str) → None

Parameters:

name (str) – the name of this registry

get(name: str) → Any

register(obj: Optional[Any] = None) → Any

Register the given object under the the name obj.__name__. Can be used as either a decorator or not. See docstring of this class for usage.

xmodaler.utils.registry.locate(name: str) → Any

Locate and return an object x using an input string {x.__module__}.{x.__qualname__}, such as “module.submodule.class_name”.

Raise Exception if it cannot be found.

class xmodaler.utils.serialize.PicklableWrapper(obj)

Bases: object

Wrap an object to make it more picklable, note that it uses heavy weight serialization libraries that are slower than pickle. It’s best to use it only on closures (which are usually not picklable).

This is a simplified version of https://github.com/joblib/joblib/blob/master/joblib/externals/loky/cloudpickle_wrapper.py

__init__(obj)