Skip to content

serve.serve_engine.configuration

EasyServeConfig dataclass

Parameters:

Name Type Description Default
host str

str: Set the host address of the server

 '0.0.0.0'
port int

int: Specify the port number that the server will run on

 2059
batch_size int

int: Set the batch size of the model

 1
max_sequence_length int

int: Set the maximum length of the text that can be generated

 4096
max_new_tokens int

int: Determine how many tokens can be added to the vocabulary

 4096
max_compile_tokens int

int: Set the maximum number of tokens that can be streamed at a time

 64
generation_ps PartitionSpec

jax.sharding.PartitionSpec : PartitionSpec to use for sharding data

 PartitionSpec('dp', 'fsdp')
temperature float

float: Control the randomness of the output

 0.1
top_p float

float: Control the diversity of the text generated

 0.95
top_k int

int: Limit the number of tokens that can be generated

 50
logging bool

bool: Print out the progress of the server

 True
mesh_axes_names Sequence[str]

Sequence[str]: Specify the names of the axes in the mesh tensor

 ('dp', 'fsdp', 'tp', 'sp')
mesh_axes_shape Sequence[int]

Sequence[int]: Specify the shape of the mesh

 (1, -1, 1, 1)
dtype str

str: Specify the data type of the model

 'fp16'
use_prefix_tokenizer bool

bool: Determine if the tokenizer should be used to generate tokens

 True
pre_compile bool

bool: Pre-compile the model

 True

Returns:

Type Description

Nothing
Source code in src/python/easydel/serve/serve_engine/configuration.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
@dataclass
class EasyServeConfig:
    """
    :param host: str: Set the host address of the server
    :param port: int: Specify the port number that the server will run on
    :param batch_size: int: Set the batch size of the model
    :param max_sequence_length: int: Set the maximum length of the text that can be generated
    :param max_new_tokens: int: Determine how many tokens can be added to the vocabulary
    :param max_compile_tokens: int: Set the maximum number of tokens that can be streamed at a time
    :param generation_ps: jax.sharding.PartitionSpec : PartitionSpec to use for sharding data
    :param temperature: float: Control the randomness of the output
    :param top_p: float: Control the diversity of the text generated
    :param top_k: int: Limit the number of tokens that can be generated
    :param logging: bool: Print out the progress of the server
    :param mesh_axes_names: Sequence[str]: Specify the names of the axes in the mesh tensor
    :param mesh_axes_shape: Sequence[int]: Specify the shape of the mesh
    :param dtype: str: Specify the data type of the model
    :param use_prefix_tokenizer: bool: Determine if the tokenizer should be used to generate tokens
    :param pre_compile: bool: Pre-compile the model
    :return: Nothing

    """
    host: str = "0.0.0.0"
    port: int = 2059

    batch_size: int = 1
    max_sequence_length: int = 4096
    max_new_tokens: int = 4096
    max_compile_tokens: int = 64
    temperature: float = 0.1
    top_p: float = 0.95
    top_k: int = 50
    repetition_penalty: float = 1.2
    greedy: bool = False

    logging: bool = True

    mesh_axes_names: Sequence[str] = ("dp", "fsdp", "tp", "sp")
    mesh_axes_shape: Sequence[int] = (1, -1, 1, 1)
    generation_ps: PartitionSpec = PartitionSpec("dp", "fsdp")
    dtype: str = "fp16"

    eos_token_id: Optional[int] = None
    pad_token_id: Optional[int] = None
    bos_token_id: Optional[int] = None

    use_prefix_tokenizer: bool = True
    pre_compile: bool = True

    verbose: bool = True

    use_mxn_break_point: bool = True

    def __repr__(self):

        """
        The __repr__ function is used to generate a string representation of an object.
        This function should return a string that can be parsed by the Python interpreter
        to recreate the object. The __repr__ function is called when you use print() on an
        object, or when you type its name in the REPL.

        :param self: Refer to the instance of the class
        :return: A string representation of the object
        """
        string = f"{self.__class__.__name__}(\n"
        for k, v in self.__dict__.items():
            if not k.startswith("_"):

                try:
                    repr_src = f"\t{k} : " + v.__str__().replace("\n", "\n\t") + "\n"
                    string += repr_src if len(repr_src) < 500 else f"\t{k} : " + f"{v.__class__.__name__}(...)" + "\n"
                except TypeError:
                    ...

        return string + ")"

    def __str__(self):

        """
        The __str__ function is called when you use the print function or when str() is used.
        It should return a string representation of the object.

        :param self: Refer to the instance of the class
        :return: The object's string representation
        """
        return self.__repr__()

__repr__()

The repr function is used to generate a string representation of an object. This function should return a string that can be parsed by the Python interpreter to recreate the object. The repr function is called when you use print() on an object, or when you type its name in the REPL.

Parameters:

Name Type Description Default
self

Refer to the instance of the class

 required

Returns:

Type Description

A string representation of the object
Source code in src/python/easydel/serve/serve_engine/configuration.py 
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
def __repr__(self):

    """
    The __repr__ function is used to generate a string representation of an object.
    This function should return a string that can be parsed by the Python interpreter
    to recreate the object. The __repr__ function is called when you use print() on an
    object, or when you type its name in the REPL.

    :param self: Refer to the instance of the class
    :return: A string representation of the object
    """
    string = f"{self.__class__.__name__}(\n"
    for k, v in self.__dict__.items():
        if not k.startswith("_"):

            try:
                repr_src = f"\t{k} : " + v.__str__().replace("\n", "\n\t") + "\n"
                string += repr_src if len(repr_src) < 500 else f"\t{k} : " + f"{v.__class__.__name__}(...)" + "\n"
            except TypeError:
                ...

    return string + ")"

__str__()

The str function is called when you use the print function or when str() is used. It should return a string representation of the object.

Parameters:

Name Type Description Default
self

Refer to the instance of the class

 required

Returns:

Type Description

The object's string representation
Source code in src/python/easydel/serve/serve_engine/configuration.py 
82
83
84
85
86
87
88
89
90
91
def __str__(self):

    """
    The __str__ function is called when you use the print function or when str() is used.
    It should return a string representation of the object.

    :param self: Refer to the instance of the class
    :return: The object's string representation
    """
    return self.__repr__()